Target detection method, device and electronic device based on large model

Abstract

This application proposes a target detection method, apparatus, and electronic device based on a large model, relating to the fields of target detection and welding technology. The method includes: inputting multimodal object data into a large model, the large model including a first encoder; performing at least one of coordinate transformation and dimension transformation on the first modality object data using the first encoder to obtain target transformation data for the first modality; fusing the target transformation data of the first modality with object data of a second modality using the first encoder to obtain fused data; and performing target detection based on the fused data using the remaining networks in the large model other than the first encoder. Therefore, multimodal data can be used for target detection, improving the robustness, generalization, and accuracy of target detection. The first encoder can be used to perform coordinate transformation and / or dimension transformation on object data of a certain modality to obtain target transformation data for that modality, thereby achieving multimodal data fusion.

Description

Technical Field

[0001] The present application relates to the field of target detection and welding technology, and in particular to a target detection method, device, electronic device and storage medium based on a large model. Background Art

[0002] Most of the target detection methods in related technologies are only applicable to certain specific scenarios. Errors in actual applications (such as tooling errors) have a great impact on detection accuracy, and they have problems such as poor robustness, poor generalization, and low accuracy. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first objective of this application is to propose a target detection method based on a large model.

[0005] The second objective of this application is to propose a target detection device based on a large model.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above-mentioned objectives, the first aspect of the present application proposes a target detection method based on a large model, including: acquiring multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object; inputting the multimodal object data into a large model, wherein the large model includes a first encoder; performing at least one of coordinate transformation and dimension transformation on the object data of the first modality by the first encoder to obtain target conversion data of the first modality; fusing the target conversion data of the first modality with the object data of the second modality by the first encoder to obtain fused data; and performing target detection based on the fused data by the remaining networks other than the first encoder in the large model.

[0010] To achieve the above-mentioned purpose, the second aspect embodiment of the present application proposes a target detection device based on a large model, including: an acquisition module, used to acquire multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object; a processing module, used to input the multimodal object data into the large model, wherein the large model includes a first encoder; a conversion module, used to perform at least one of coordinate conversion and dimension conversion on the object data of the first modality through the first encoder to obtain target conversion data of the first modality; a fusion module, used to fuse the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data; a detection module, used to perform target detection based on the fused data through the remaining networks other than the first encoder in the large model.

[0011] To achieve the above-mentioned purpose, the third aspect embodiment of the present application proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect embodiment above.

[0012] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment above.

[0013] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect of the present application.

[0014] The large-model-based target detection method, device, electronic device and storage medium provided in the present application can use multimodal data for target detection, thereby improving the robustness, generalization and accuracy of target detection. The first encoder can be used to perform at least one of coordinate transformation and dimension transformation on object data of a certain modality to obtain target conversion data of the modality to achieve multimodal data fusion, and target detection is performed based on the fused data through the remaining networks of the large model, thereby simplifying user operations, improving target detection efficiency, and optimizing the user experience in target detection scenarios, and is suitable for weld detection scenarios.

[0015] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A schematic diagram of a flow chart of a target detection method based on a large model provided in an embodiment of the present application;

[0018] Figure 2 A schematic diagram of a flow chart of another large model-based target detection method provided in an embodiment of the present application;

[0019] Figure 3 A schematic diagram of a flow chart of another large model-based target detection method provided in an embodiment of the present application;

[0020] Figure 4 A schematic diagram of a large model provided in an embodiment of the present application;

[0021] Figure 5 A schematic diagram of a large model-based target detection method provided in an embodiment of the present application;

[0022] Figure 6 A flowchart of a model training method provided in an embodiment of the present application;

[0023] Figure 7 A schematic diagram of the structure of a large model-based target detection device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0025] The following describes the large model-based target detection method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0026] Figure 1 A flowchart of a large model-based target detection method provided in an embodiment of the present application.

[0027] Most of the target detection methods in related technologies are only applicable to certain specific scenarios. Errors in actual applications (such as tooling errors) have a great impact on detection accuracy, and they have problems such as poor robustness, poor generalization, and low accuracy.

[0028] To address this problem, an embodiment of the present application provides a target detection method based on a large model, which can use multimodal data for target detection, thereby improving the robustness, generalization and accuracy of target detection. A first encoder can be used to perform at least one of coordinate transformation and dimension transformation on object data of a certain modality to obtain target conversion data of the modality to achieve multimodal data fusion, and target detection is performed based on the fused data through the remaining networks of the large model, thereby simplifying user operations, improving target detection efficiency, and optimizing the user experience in target detection scenarios, and is suitable for weld detection scenarios.

[0029] like Figure 1 As shown, the method comprises the following steps:

[0030] S101, acquiring multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object.

[0031] It should be noted that there are no excessive restrictions on the objects. For example, in the weld inspection scenario, the objects may include workpieces, in the autonomous driving scenario, the objects may include roads, and in the security monitoring scenario, the objects may include factories, offices, etc. There are no excessive restrictions on the first image data. For example, it may include RGB images, depth images, thermal imaging images, multi-spectral images, multi-spectral images, etc. In addition to the first image data and the first point cloud data, the multi-modal object data may also include object data of other modalities, such as radar data, etc.

[0032] Exemplarily, acquiring multimodal object data may include acquiring first image data and first point cloud data through a 3D (Dimensional) camera.

[0033] Exemplarily, acquiring multimodal object data may include acquiring first image data through an RGB camera and acquiring first point cloud data through a laser scanning device.

[0034] Most of the target detection methods in related technologies are only applicable to certain specific scenarios. Errors in actual applications (such as tooling errors) have a great impact on detection accuracy, and they have problems such as poor robustness, poor generalization, and low accuracy.

[0035] For example, taking the weld detection scenario as an example, in the related technology, the intersection line of the two planes can be calculated by searching for a plane or a predefined standard arc surface in the workpiece point cloud to extract the weld information. This method requires a predefined surface formula or normal vector threshold when calculating the surface, and is only applicable to specific scenarios, such as steel structure corner welding. In addition, if there are tooling errors such as protrusions, depressions, and dirt on the surface of the workpiece, the accuracy of the weld detection results will be low, that is, the generalization ability of this method is poor. In addition, this method operates directly in the 3D point cloud, and the efficiency is also low, often taking tens of seconds.

[0036] In this scheme, multimodal data can be used for target detection, that is, at least image data and point cloud data are used for target detection. Image data carries semantic information, and point cloud data carries spatial information. Therefore, target detection can be performed by comprehensively considering semantic information and spatial information. This scheme is suitable for a variety of target detection scenarios. The error in actual application has little effect on the detection accuracy, which improves the robustness, generalization and accuracy of target detection.

[0037] Especially in the weld detection scenario, this solution can comprehensively consider semantic information and spatial information for weld detection, and is suitable for a variety of weld detection scenarios. The error in actual application has little effect on the detection accuracy. For example, if there are tooling errors such as protrusions, depressions, and dirt on the workpiece surface, the detection accuracy of this solution is still high, which improves the robustness, generalization and accuracy of target detection.

[0038] In addition, the present solution can take spatial information into consideration for weld detection, which helps to distinguish welds in different spatial positions (such as horizontal, vertical, vertical, and overhead), as well as multiple welds that are adhered.

[0039] S102, inputting multimodal object data into a large model, wherein the large model includes a first encoder.

[0040] It should be noted that a large model refers to a machine learning model with a large parameter scale and complexity, which requires a large amount of computing resources and storage space for training and storage, and often requires distributed computing and special hardware acceleration technology. Large models have stronger generalization and expression capabilities. The large model can be implemented using any large model in the relevant technology, and there is no excessive limitation here. For example, it can be a Transformer model. It should be noted that the Transformer model is a neural network model based on the self-attention mechanism.

[0041] Exemplarily, the first encoder is a patch embedding network.

[0042] The target detection methods in related technologies have complex user operations and low target detection efficiency, resulting in poor user experience in target detection scenarios. In this solution, a large model can be used for target detection, which simplifies user operations, improves target detection efficiency, and optimizes user experience in target detection scenarios. In addition, as data continues to accumulate, the large model can be continuously iterated and optimized, which helps to improve target detection accuracy.

[0043] Especially in the weld detection scenario, this solution only takes 500ms (milliseconds) to identify one time, which improves the efficiency of weld detection and optimizes the user experience in the weld detection scenario.

[0044] S103, performing at least one of coordinate conversion and dimension conversion on the object data of the first modality through the first encoder to obtain target conversion data of the first modality.

[0045] It should be noted that the multimodal data includes object data of the first modality and object data of the second modality. There are no excessive restrictions on the specific methods of coordinate transformation and dimension transformation. The first modality and the second modality are different modalities. The coordinate system and dimension of the target conversion data of the first modality are consistent with the object data of the second modality, that is, the coordinate system of the target conversion data of the first modality is consistent with the coordinate system of the object data of the second modality (that is, the second coordinate system hereinafter), and the dimension of the target conversion data of the first modality is consistent with the dimension of the object data of the second modality.

[0046] In an embodiment of the present application, the first encoder performs at least one of coordinate conversion and dimension conversion on the object data of the first modality to obtain the target conversion data of the first modality, including the following possible implementations:

[0047] Method 1: In response to the first coordinate system of the object data of the first modality being consistent with the second coordinate system of the object data of the second modality, and the dimension of the object data of the first modality being inconsistent with the dimension of the object data of the second modality, the object data of the first modality is dimensionally converted by a first encoder to obtain third converted data as the target conversion data of the first modality, wherein the dimension of the third converted data is consistent with the dimension of the object data of the second modality.

[0048] Therefore, when the coordinate systems of the object data of the two modes are consistent but their dimensions are inconsistent, the first encoder can be used to perform dimension conversion on the object data of the first mode to obtain the target conversion data of the first mode. In this case, only dimension conversion is required.

[0049] Method 2: In response to the fact that a first coordinate system of object data of a first modality is inconsistent with a second coordinate system of object data of a second modality, and a dimension of the object data of the first modality is consistent with a dimension of the object data of the second modality, coordinate transformation is performed on the object data of the first modality by a first encoder to obtain first transformed data in a second coordinate system as target transformed data of the first modality.

[0050] Therefore, when the coordinate systems of the object data of the two modes are inconsistent and their dimensions are consistent, the first encoder can be used to perform coordinate transformation on the object data of the first mode to obtain the target transformation data of the first mode. In this case, only coordinate transformation is required.

[0051] Method 3: Perform coordinate conversion on the first point cloud data through the first encoder to obtain second point cloud data in the camera coordinate system, and perform dimension conversion on the second point cloud data through the first encoder to obtain third point cloud data as the target conversion data of the point cloud modality, wherein the dimension of the third point cloud data is consistent with the dimension of the first image data.

[0052] It can be understood that the coordinate system of the image data is the camera coordinate system, and the coordinate systems of the first point cloud data and the image data are inconsistent, and their dimensions are also inconsistent.

[0053] Therefore, the first encoder can be used to convert the point cloud data into the camera coordinate system, and then the converted point cloud data can be dimensionally converted, that is, the first encoder is used to perform coordinate conversion and dimension conversion on the point cloud data to obtain the target conversion data of the point cloud modality, so that the coordinate system and dimension of the final point cloud data are consistent with the image data, so as to facilitate the subsequent fusion of the point cloud data and the image data.

[0054] Exemplarily, the coordinate system of the first point cloud data is a world coordinate system, and the first point cloud data is converted by the first encoder to obtain the second point cloud data in the camera coordinate system, which can be achieved by the following formula:

[0055]

[0056] in, is the second point cloud data, which is a vector of 3 rows and 1 column, representing the coordinates of the 3D point in the camera coordinate system. is the first point cloud data, which is a vector of 3 rows and 1 column, representing the coordinates of the 3D point in the world coordinate system. is a 3-row 3-column rotation matrix, representing the rotation operation from the world coordinate system to the camera coordinate system. A translation vector with 3 rows and 1 column, representing the translation operation from the world coordinate system to the camera coordinate system.

[0057] Exemplarily, the dimension of the first image data is [b, c, H, W], where b is the batch size, i.e., the number of images processed at one time, c is the number of channels of the image, H is the height of the entire image, and W is the width of the entire image. For example, taking an RGB image as an example, an RGB image includes three channels: R (red), G (green), and B (blue), and c=3 at this time.

[0058] The dimension of the second point cloud data is [b, M, 3], where b is the batch size and M is the number of 3D points. 3D points can be represented by three-dimensional coordinates (such as x, y, and z coordinates). That is, the second point cloud data includes the three-dimensional coordinates of each of the M 3D points. For example, the second point cloud data includes three channels: x, y, and z.

[0059] Performing dimension conversion on the second point cloud data through the first encoder to obtain the third point cloud data may include filling the second point cloud data into a tensor with a dimension of [b, c, H, W] through the first encoder, and using the filled tensor with a dimension of [b, c, H, W] as the third point cloud data.

[0060] For example, data of different channels of the second point cloud data are filled into different channels of a tensor of dimension [b, c, H, W].

[0061] For example, filling the second point cloud data into a tensor of dimension [b, c, H, W] through the first encoder includes filling the data of the jth channel of the second point cloud data into the jth channel of the tensor of dimension [b, c, H, W] through the first encoder, where j is a positive integer not greater than M.

[0062] For example, the second point cloud data is filled into a tensor of dimension [b, c, H, W] through the first encoder, including projecting the jth 3D point to the image plane through the first encoder to obtain the pixel position of the jth 3D point, and filling the three-dimensional coordinates of the jth 3D point into the pixel position of the jth 3D point in the tensor of dimension [b, c, H, W] through the first encoder.

[0063] For example, the second point cloud data includes the x, y, and z coordinates of 3D point 1, and the pixel position of 3D point 1 is (H1, W1). The x coordinate of 3D point 1 can be filled into the tensor of dimension [b, c, H, W] with the pixel position (H1, W1) and the data point of the first channel through the first encoder.

[0064] The first encoder fills the y coordinate of 3D point 1 into the tensor of dimension [b,c,H,W] with the pixel position (H1,W1) and the data point of the second channel.

[0065] The first encoder fills the z coordinate of 3D point 1 into the tensor of dimension [b,c,H,W] with the pixel position (H1,W1) and the data point of the third channel.

[0066] If a data point in the tensor of dimension [b, c, H, W] does not have the second point cloud data that can be filled, the data point can be filled with 0, that is, the missing part can be filled with 0.

[0067] S104, fusing the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data.

[0068] It should be noted that there are no excessive restrictions on the specific method of data fusion.

[0069] For example, multimodal object data includes object data of modality A, object data of modality B, and object data of modality C.

[0070] The first encoder may perform at least one of coordinate conversion and dimension conversion on the object data of modality A to obtain the target conversion data of modality A, the first encoder may perform at least one of coordinate conversion and dimension conversion on the object data of modality B to obtain the target conversion data of modality B, and the first encoder may fuse the target conversion data of modality A, the target conversion data of modality B, and the object data of modality C to obtain fused data. It can be understood that in this embodiment, modalities A and B are the first modality, modality C is the second modality, the coordinate system and dimension of the target conversion data of modality A are consistent with the object data of modality C, and the coordinate system and dimension of the target conversion data of modality B are consistent with the object data of modality C.

[0071] The first encoder may perform at least one of coordinate conversion and dimension conversion on the object data of modality A to obtain the target conversion data of modality A, the first encoder may perform at least one of coordinate conversion and dimension conversion on the object data of modality C to obtain the target conversion data of modality C, and the first encoder may fuse the target conversion data of modality A, the target conversion data of modality C, and the object data of modality B to obtain the fused data. It can be understood that in this embodiment, modalities A and C are the first modality, modality B is the second modality, the coordinate system and dimension of the target conversion data of modality A are consistent with the object data of modality B, and the coordinate system and dimension of the target conversion data of modality C are consistent with the object data of modality B.

[0072] In the embodiment of the present application, the target conversion data of the first modality and the object data of the second modality are merged by the first encoder, including the following possible implementations:

[0073] Method 1: extract features from the first image data through the first encoder to obtain a first feature, extract features from the third point cloud data through the first encoder to obtain a second feature, and splice the first feature and the second feature through the first encoder to obtain fused data.

[0074] Therefore, the first encoder can be used to extract features from the first image data and the third point cloud data respectively to obtain the first feature and the second feature, and the first feature and the second feature can be spliced ​​to achieve the fusion of the image data and the point cloud data.

[0075] It should be noted that there is no excessive limitation on the first feature, for example, it may include image features, semantic features, etc. It should be noted that image features are used to describe the basic visual elements in the image, and may include color features, texture features, shape features, and spatial relationship features, etc. Semantic features are used to describe the image content, and may include the category, action, posture, emotion, etc. of the instance in the image. Terms such as instance, entity, and target are interchangeable.

[0076] It should be noted that there are no excessive restrictions on the second feature, for example, it may include a spatial feature.

[0077] Exemplarily, feature extraction is performed on the first image data by the first encoder to obtain the first feature, including segmenting the first image data by the first encoder to obtain second image data of N image blocks, where N is a positive integer, and feature extraction is performed on the second image data of the i-th image block by the first encoder to obtain the first feature of the i-th image block, where i is a positive integer not greater than N.

[0078] Exemplarily, the third point cloud data is subjected to feature extraction by the first encoder to obtain the second feature, including segmenting the third point cloud data by the first encoder to obtain the fourth point cloud data of each of the N image blocks, and extracting the fourth point cloud data of the i-th image block by the first encoder to obtain the second feature of the i-th image block.

[0079] For example, the dimensions of the first image data and the third point cloud data are both [b, c, H, W], and the dimensions of the second image data and the fourth point cloud data are both [b, h, w, c p1 p2], where h is the number of image blocks divided in the height direction, w is the number of image blocks divided in the width direction, p1 is the height of the image block, and p2 is the width of the image block.

[0080] For example, the method further includes normalizing the first feature to update the first feature, and normalizing the second feature to update the second feature.

[0081] Exemplarily, the first feature and the second feature are spliced ​​by the first encoder to obtain fused data, including splicing the first feature of the i-th image block and the second feature of the i-th image block by the first encoder to obtain fused data of the i-th image block.

[0082] Method 2: splicing the target conversion data of the first modality and the object data of the second modality through the first encoder to obtain spliced ​​data as fused data.

[0083] Method 3: splicing the target conversion data of the first modality and the object data of the second modality through the first encoder to obtain spliced ​​data, and performing feature extraction on the spliced ​​data through the first encoder to obtain fused data.

[0084] S105, performing target detection based on the fused data through the remaining networks except the first encoder in the large model.

[0085] It should be noted that there are no excessive restrictions on the target detection results, for example, they may include the segmentation mask, classification results and bounding box corresponding to the first image data. The segmentation mask refers to the segmentation mask of the instance in the image. The terms such as segmentation mask and mask are interchangeable. There are no excessive restrictions on the segmentation mask, for example, they may include arrays, multi-value images (such as binary images), grayscale images, etc. There are no excessive restrictions on the classification results, for example, they may include the category of the instance in the image, the category of the pixel in the image, the probability that the pixel belongs to the target category, etc. The bounding box refers to the bounding box of the instance in the image.

[0086] Exemplarily, taking the weld detection scenario as an example, the target detection result is the weld detection result, and the weld detection result includes the segmentation mask of the weld in the image, whether the pixel in the image is a weld, the probability that the pixel belongs to the weld, the bounding box of the weld in the image, the weld trajectory, etc.

[0087] In one embodiment, the method further includes planning a route for a welding robot based on the weld detection result. For example, the route planning for the welding robot can be performed based on the weld trajectory. Thus, in the weld detection scenario, the route planning for the welding robot can also be performed based on the weld detection result to guide the robot to perform precise welding, which is suitable for the robot intelligent welding scenario.

[0088] In summary, according to the target detection method based on the large model of the embodiment of the present application, multimodal object data is obtained, wherein the multimodal object data includes the first image data of the object and the first point cloud data of the object, and the multimodal object data is input into the large model, wherein the large model includes a first encoder, and the first encoder performs at least one of coordinate conversion and dimension conversion on the object data of the first modality to obtain the target conversion data of the first modality, and the first encoder fuses the target conversion data of the first modality with the object data of the second modality to obtain fused data, and the remaining networks other than the first encoder in the large model perform target detection based on the fused data. Thus, multimodal data can be used for target detection, which improves the robustness, generalization and accuracy of target detection, and the first encoder can be used to perform at least one of coordinate conversion and dimension conversion on the object data of a certain modality to obtain the target conversion data of the modality to achieve multimodal data fusion, and the remaining networks of the large model perform target detection based on the fused data, which simplifies user operation, improves target detection efficiency, and optimizes user experience in target detection scenarios, and is suitable for weld detection scenarios.

[0089] An embodiment of the present application provides another target detection method based on a large model.

[0090] like Figure 2 As shown, the method may include the following steps:

[0091] S201, acquiring multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object.

[0092] S202, inputting multimodal object data into a large model, wherein the large model includes a first encoder.

[0093] For the relevant contents of steps S201 - S202 , please refer to the above embodiment and will not be described again here.

[0094] S203, in response to the inconsistency between the first coordinate system of the object data of the first modality and the second coordinate system of the object data of the second modality, coordinate conversion is performed on the object data of the first modality by using a first encoder to obtain first conversion data in the second coordinate system.

[0095] S204, in response to the dimensionality of the first conversion data being inconsistent with the dimensionality of the object data of the second modality, the first conversion data is dimensionally converted by a first encoder to obtain second conversion data as the target conversion data of the first modality, wherein the dimensionality of the second conversion data is consistent with the dimensionality of the object data of the second modality.

[0096] For example, multimodal object data includes object data of modality A and object data of modality B. Modality A is a first modality, and modality B is a second modality.

[0097] In response to the inconsistency between the first coordinate system of the object data of modality A and the second coordinate system of the object data of modality B, the object data of modality A is coordinate-converted by a first encoder to obtain first conversion data in the second coordinate system.

[0098] In response to the inconsistency between the dimension of the first conversion data and the dimension of the object data of modality B, the first conversion data is dimensionally converted by the first encoder to obtain second conversion data as the target conversion data of modality A, wherein the dimension of the second conversion data is consistent with the dimension of the object data of modality B.

[0099] Exemplarily, the first encoder performs dimension conversion on the first conversion data to obtain the second conversion data, including, in response to the dimension of the first conversion data being lower than the dimension of the object data of the second modality, performing dimension increase processing on the first conversion data by the first encoder to obtain the second conversion data. It should be noted that there is no excessive limitation on the dimension increase method, for example, it may include feature construction, polynomial features, encoding, etc.

[0100] Exemplarily, the first encoder performs dimension conversion on the first conversion data to obtain the second conversion data, including, in response to the dimension of the first conversion data being higher than the dimension of the object data of the second modality, performing dimension reduction processing on the first conversion data by the first encoder to obtain the second conversion data. It should be noted that there is no excessive limitation on the dimension reduction method, for example, it may include PCA (Principal Component Analysis), Laplace eigenmap, etc.

[0101] S205 , in response to the dimension of the first conversion data being consistent with the dimension of the object data of the second modality, using the first encoder to take the first conversion data as the target conversion data of the first modality.

[0102] For example, multimodal object data includes object data of modality A and object data of modality B. Modality A is a first modality, and modality B is a second modality.

[0103] In response to the inconsistency between the first coordinate system of the object data of modality A and the second coordinate system of the object data of modality B, the object data of modality A is coordinate-converted by a first encoder to obtain first conversion data in the second coordinate system.

[0104] In response to the dimension of the first conversion data being consistent with the dimension of the object data of modality B, the first conversion data is taken as the target conversion data of modality A by the first encoder.

[0105] S206, fusing the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data.

[0106] S207, performing target detection based on the fused data through the remaining networks except the first encoder in the large model.

[0107] For the relevant contents of steps S206-S207, please refer to the above embodiment and will not be repeated here.

[0108] In summary, according to the large model-based target detection method of the embodiment of the present application, when the coordinate systems of the object data of the two modalities are inconsistent, the first encoder can be used to perform coordinate transformation on the object data of the first modality to obtain first transformation data. When the dimensions of the first transformation data and the object data of the second modality are consistent, the first transformation data can be directly used as the target transformation data of the first modality. Alternatively, when the dimensions of the first transformation data and the object data of the second modality are inconsistent, the first encoder can be used to perform dimension transformation on the first transformation data to obtain the target transformation data of the first modality.

[0109] An embodiment of the present application provides another target detection method based on a large model.

[0110] like Figure 3 As shown, the method may include the following steps:

[0111] S301, acquiring multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object.

[0112] S302, inputting multimodal object data into a large model, wherein the large model includes a first encoder.

[0113] For the relevant contents of steps S301 - S302 , please refer to the above embodiments and will not be described again here.

[0114] S303, in response to the dimensionality of the object data of the first modality being inconsistent with the dimensionality of the object data of the second modality, the object data of the first modality is dimensionally converted by a first encoder to obtain third converted data, wherein the dimensionality of the third converted data is consistent with the dimensionality of the object data of the second modality.

[0115] S304, in response to the inconsistency between the third coordinate system of the third conversion data and the second coordinate system of the object data of the second modality, coordinate conversion is performed on the third conversion data by the first encoder to obtain fourth conversion data in the second coordinate system as target conversion data of the first modality.

[0116] For example, multimodal object data includes object data of modality A and object data of modality B. Modality A is a first modality, and modality B is a second modality.

[0117] In response to the inconsistency between the dimension of the object data of modality A and the dimension of the object data of modality B, the object data of modality A is dimensionally converted by a first encoder to obtain third converted data, wherein the dimension of the third converted data is consistent with the dimension of the object data of modality B.

[0118] In response to the inconsistency between the third coordinate system of the third conversion data and the second coordinate system of the object data of modality B, the third conversion data is coordinate-converted by the first encoder to obtain fourth conversion data in the second coordinate system as the target conversion data of modality A.

[0119] It should be noted that the relevant contents of coordinate conversion and dimension conversion can be found in the above embodiments and will not be repeated here.

[0120] S305 , in response to the third coordinate system being consistent with the second coordinate system, using the first encoder to use the third conversion data as target conversion data of the first modality.

[0121] For example, multimodal object data includes object data of modality A and object data of modality B. Modality A is a first modality, and modality B is a second modality.

[0122] In response to the inconsistency between the dimension of the object data of modality A and the dimension of the object data of modality B, the object data of modality A is dimensionally converted by a first encoder to obtain third converted data, wherein the dimension of the third converted data is consistent with the dimension of the object data of modality B.

[0123] In response to the third coordinate system of the third conversion data being consistent with the second coordinate system of the object data of the modality B, the third conversion data is taken as the target conversion data of the modality A by the first encoder.

[0124] S306, fusing the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data.

[0125] S307, performing target detection based on the fused data through the remaining networks except the first encoder in the large model.

[0126] For the relevant contents of steps S306-S307, please refer to the above embodiment and will not be repeated here.

[0127] In summary, according to the large model-based target detection method of the embodiment of the present application, when the dimensions of the object data of the two modalities are inconsistent, the first encoder can be used to perform dimension conversion on the object data of the first modality to obtain third conversion data. When the coordinate systems of the third conversion data and the object data of the second modality are consistent, the third conversion data can be directly used as the target conversion data of the first modality. Alternatively, when the coordinate systems of the third conversion data and the object data of the second modality are inconsistent, the first encoder can be used to perform coordinate conversion on the third conversion data to obtain the target conversion data of the first modality.

[0128] Based on any of the above embodiments, Figure 4 As shown, the large model also includes a feature extraction network, a decoder, a second encoder and an MLP (Multilayer Perceptron).

[0129] It should be noted that there are no excessive restrictions on the feature extraction network, such as Figure 4 As shown, the feature extraction network includes a backbone network, a position encoder and a third encoder. Exemplarily, the backbone network is ResNet50, which is a deep convolutional neural network, the position encoder is a position embedding network, and the third encoder is a 6-layer encoding network, for example, it can be composed of 6 encoders connected in series.

[0130] Exemplarily, the decoder is a 9-layer decoding network, for example, it may be composed of 9 decoders connected in series.

[0131] Exemplarily, the second encoder is a pixel embedding network.

[0132] An embodiment of the present application provides another target detection method based on a large model.

[0133] like Figure 5 As shown, the method may include the following steps:

[0134] S501, acquiring multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object.

[0135] S502, inputting multimodal object data into a large model, wherein the large model includes a first encoder.

[0136] S503, performing at least one of coordinate conversion and dimension conversion on the object data of the first modality through the first encoder to obtain target conversion data of the first modality.

[0137] S504: fusing the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data.

[0138] For the relevant contents of steps S501 - S504 , please refer to the above embodiment and will not be described again here.

[0139] S505, extracting features from the fused data through a feature extraction network to obtain a third feature.

[0140] In one embodiment, feature extraction is performed on the fused data through a feature extraction network to obtain a third feature, including performing multi-scale feature extraction on the fused data through a feature extraction network to obtain a multi-scale feature as the third feature.

[0141] In one embodiment, the feature extraction network includes a backbone network, a position encoder and a third encoder. The feature extraction network performs feature extraction on the fused data to obtain the third feature, including performing feature extraction on the fused data through the backbone network to obtain the sixth feature, performing position encoding on the sixth feature through the position encoder to obtain position encoding data of the sixth feature, and performing feature extraction on the sixth feature and the position encoding data through the third encoder to obtain the third feature. Thus, the position encoder can be used to obtain the position encoding data of the sixth feature, so that the third encoder can understand the position information of the elements contained in the sixth feature to assist the third encoder in feature extraction, thereby improving the accuracy of feature extraction.

[0142] S506, performing target detection based on the third feature through the decoder to obtain a fourth feature.

[0143] S507, performing feature extraction on the third feature and the fused data through a second encoder to obtain a fifth feature.

[0144] S508: Obtain a segmentation mask corresponding to the first image data based on the fourth feature and the fifth feature through a multi-layer perceptron.

[0145] In one embodiment, a segmentation mask corresponding to the first image data is obtained based on the fourth feature and the fifth feature through a multilayer perceptron, including extracting the fourth feature through a multilayer perceptron to obtain a seventh feature, and calculating the dot product between the fifth feature and the seventh feature through the multilayer perceptron as the segmentation mask corresponding to the first image data.

[0146] Exemplarily, obtaining a segmentation mask corresponding to the first image data based on the fourth feature and the fifth feature through a multilayer perceptron can be implemented by the following formula:

[0147] mask_embed = self.mask_embed(decoder_output)

[0148] outputs_mask = torch.einsum("bqc,bchw->bqhw", mask_embed, mask_features)

[0149] Among them, decoder_output is the output data of the decoder, that is, the fourth feature, self.mask_embed ( ) is the function corresponding to MLP, mask_embed is the seventh feature, mask_features is the fifth feature, and outputs_mask is the segmentation mask.

[0150] torch.einsum( ) is a function used to perform the Einstein summation convention. bqc means that the mask_embed tensor has three dimensions, namely the batch size (b), the number of categories (query) (q), and the number of channels (c). bchw means that the mask_features tensor has four dimensions, namely the batch size (b), the number of channels (c), the height (h), and the width (w). bqhw means that the output tensor has four dimensions, namely the batch size (b), the number of categories (q), the height (h), and the width (w).

[0151] In this example, torch.einsum( ) function calculates the dot product between mask_embed and mask_features and obtains a tensor outputs_mask with dimension (b, q, h, w).

[0152] S509: Obtain a classification result corresponding to the first image data based on the fourth feature through a multilayer perceptron.

[0153] S510: Obtain a bounding box corresponding to the first image data based on the fourth feature through a multilayer perceptron.

[0154] It should be noted that there are no excessive restrictions on the execution order of steps S508 - S510 , for example, they can be executed serially or in parallel.

[0155] For example, classification results, segmentation masks, and bounding boxes can be obtained through different multi-layer perceptrons. Therefore, different multi-layer perceptrons can be used to perform image classification tasks, bounding box generation tasks, and mask generation tasks respectively, which has the advantage of high efficiency compared to the related art of using the same network to perform image classification tasks, bounding box generation tasks, and mask generation tasks.

[0156] For example, Figure 4 As shown, the segmentation mask corresponding to the first image data can be obtained by multilayer perceptron 1 based on the fourth feature and the fifth feature, the classification result corresponding to the first image data can be obtained by multilayer perceptron 2 based on the fourth feature, and the bounding box corresponding to the first image data can be obtained by multilayer perceptron 3 based on the fourth feature.

[0157] In summary, according to the large model-based target detection method of the embodiment of the present application, the fourth feature and the fifth feature can be comprehensively considered through a multi-layer perceptron to perform mask generation, thereby improving the accuracy of the segmentation mask, and the fourth feature can be considered through a multi-layer perceptron to perform image classification and bounding box generation respectively.

[0158] An embodiment of the present application provides a model training method.

[0159] like Figure 6 As shown, the method may include the following steps:

[0160] S601, obtaining a first loss function of the large model based on the predicted classification result corresponding to the sample image data.

[0161] It should be noted that there are no excessive restrictions on the first loss function, for example, it may include the focal loss (FocalLoss) function, CE (Cross Entropy), MSE (Mean-Square Error), KL (Kullback-Leibler) divergence, contrast loss function, etc.

[0162] In one embodiment, based on the predicted classification result corresponding to the sample image data, a first loss function of the large model is obtained, including obtaining the sample classification result corresponding to the sample image data, and obtaining the first loss function based on the predicted classification result and the sample classification result.

[0163] For example, taking the first loss function as a caustic loss function as an example, the first loss function as follows:

[0164]

[0165] in, is the predicted probability that the pixel in the sample image belongs to the target category, is a balancing factor used to adjust the weight ratio between positive and negative samples. It is an adjustment factor used to control the weight of difficult-to-classify samples.

[0166] S602, obtaining a second loss function of the large model based on the predicted bounding box corresponding to the sample image data.

[0167] It should be noted that there are no excessive restrictions on the second loss function, for example, it may include IoU (Intersection over Union) loss function, caustic loss function, CE, MSE, KL divergence, contrast loss function, etc.

[0168] In one embodiment, based on the predicted bounding box corresponding to the sample image data, a second loss function of the large model is obtained, including obtaining a sample bounding box corresponding to the sample image data, and obtaining a second loss function based on the predicted bounding box and the sample bounding box.

[0169] For example, taking the second loss function as the IoU loss function, the second loss function as follows:

[0170]

[0171] in, To predict the bounding box, is the sample bounding box, for and The intersection area between for and The union area between .

[0172] S603, obtaining a third loss function of the large model based on the predicted segmentation mask corresponding to the sample image data.

[0173] It should be noted that there are no excessive restrictions on the third loss function, for example, it may include caustic loss function, Dice loss function, IoU loss function, CE, MSE, KL divergence, contrast loss function, etc.

[0174] In one embodiment, based on the predicted segmentation mask corresponding to the sample image data, a third loss function of the large model is obtained, including obtaining a sample segmentation mask corresponding to the sample image data, and obtaining a third loss function based on the predicted segmentation mask and the sample segmentation mask.

[0175] In one embodiment, a third loss function of the large model is obtained based on the predicted segmentation mask corresponding to the sample image data, including obtaining the seventh loss function of the large model based on the predicted probability that the predicted segmentation mask belongs to the target category, obtaining the eighth loss function of the large model based on the predicted segmentation mask and the sample segmentation mask, and obtaining the third loss function based on at least one of the seventh loss function and the eighth loss function.

[0176] For example, taking the seventh loss function as the caustic loss function as an example, the seventh loss function as follows:

[0177]

[0178] in, The predicted probability that the segmentation mask belongs to the target class.

[0179] For example, taking the eighth loss function as the Dice loss function, the eighth loss function as follows:

[0180]

[0181] in, To predict the segmentation mask, is the sample segmentation mask, for and The intersection area between for The area of for area.

[0182] Exemplarily, based on at least one of the seventh loss function and the eighth loss function, a third loss function is obtained, including using the seventh loss function as the third loss function, or using the eighth loss function as the third loss function, or performing weighted summation of the seventh loss function and the eighth loss function to obtain the third loss function.

[0183] S604, obtaining a predicted target detection result corresponding to the sample image data based on at least one of a predicted classification result, a predicted bounding box, and a predicted segmentation mask corresponding to the sample image data.

[0184] S605, based on the predicted target detection result, obtain the fourth loss function of the large model.

[0185] It should be noted that there are no excessive restrictions on the fourth loss function, for example, it may include caustic loss function, Dice loss function, IoU loss function, CE, MSE, KL divergence, contrast loss function, etc.

[0186] In one embodiment, based on the predicted target detection results, a fourth loss function of the large model is obtained, including obtaining sample target detection results corresponding to the sample image data, and obtaining a fourth loss function based on the predicted target detection results and the sample target detection results.

[0187] In one embodiment, based on the predicted target detection result, a fourth loss function of the large model is obtained, including a fifth loss function of the large model based on the number of breakpoints contained in the predicted weld trajectory, a sixth loss function of the large model based on the target distance between two adjacent breakpoints contained in the predicted weld trajectory, and a fourth loss function based on at least one of the fifth loss function and the sixth loss function. Thus, in the weld detection scenario, the fourth loss function can be obtained by considering the number of breakpoints contained in the predicted weld trajectory and / or the target distance between the breakpoints, which helps to improve the continuity of the weld trajectory.

[0188] It should be noted that the target distance refers to the distance between two adjacent fracture points contained in the predicted weld trajectory.

[0189] Exemplarily, the fifth loss function as follows:

[0190]

[0191] in, To predict the number of fracture points contained in the weld trajectory, The number of connected segments contained in the predicted weld trajectory.

[0192] In some examples, based on the target distance between two adjacent fracture points included in the predicted weld trajectory, a sixth loss function of the large model is obtained, including obtaining the sum of multiple target distances corresponding to the predicted weld trajectory as the sixth loss function.

[0193] Exemplarily, the sixth loss function as follows:

[0194]

[0195] in, To predict the weld trajectory, connected segments, To predict the weld trajectory, connected segments, for The position point on for The position point on is the Euclidean distance between two adjacent breakpoints.

[0196] It should be noted that the relevant content of the fourth loss function is obtained based on at least one of the fifth loss function and the sixth loss function. The relevant content of the third loss function is obtained based on at least one of the seventh loss function and the eighth loss function in the above embodiment, which will not be repeated here.

[0197] S606, training the large model based on at least two of the first loss function, the second loss function, the third loss function and the fourth loss function.

[0198] In one embodiment, the large model is trained based on at least two of the first loss function, the second loss function, the third loss function, and the fourth loss function, including obtaining a total loss function of the large model based on at least two of the first loss function, the second loss function, the third loss function, and the fourth loss function, and training the large model based on the total loss function.

[0199] It should be noted that, for relevant content about the total loss function of the large model obtained based on at least two of the first loss function, the second loss function, the third loss function and the fourth loss function, please refer to the relevant content about the third loss function obtained based on at least one of the seventh loss function and the eighth loss function in the above embodiment, which will not be repeated here.

[0200] In summary, according to the model training method of the embodiment of the present application, at least two of the first loss function, the second loss function, the third loss function and the fourth loss function can be comprehensively considered to train the large model, that is, at least two of the classification loss, bounding box loss, segmentation mask loss and detection result loss can be comprehensively considered to jointly optimize the large model. The loss functions of different tasks can jointly guide the learning process of the large model, which helps to improve the accuracy of the large model in image classification tasks, bounding box generation tasks and mask generation tasks.

[0201] In order to implement the above embodiments, the present application also proposes a target detection device based on a large model.

[0202] Figure 7 A schematic diagram of the structure of a large model-based target detection device provided in an embodiment of the present application.

[0203] like Figure 7 As shown, the large model-based target detection device 100 includes: an acquisition module 110, a processing module 120, a conversion module 130, a fusion module 140 and a detection module 150.

[0204] An acquisition module 110, configured to acquire multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object;

[0205] A processing module 120, configured to input the multimodal object data into a large model, wherein the large model comprises a first encoder;

[0206] The conversion module 130 is used to perform at least one of coordinate conversion and dimension conversion on the object data of the first modality through the first encoder to obtain target conversion data of the first modality;

[0207] A fusion module 140, configured to fuse the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data;

[0208] The detection module 150 is used to perform target detection based on the fused data through the remaining networks other than the first encoder in the large model.

[0209] Further, in a possible implementation of the embodiment of the present application, the conversion module 130 is also used for: in response to the first coordinate system of the object data of the first modality being inconsistent with the second coordinate system of the object data of the second modality, performing coordinate conversion on the object data of the first modality through the first encoder to obtain first conversion data in the second coordinate system; in response to the dimension of the first conversion data being inconsistent with the dimension of the object data of the second modality, performing dimension conversion on the first conversion data through the first encoder to obtain second conversion data as target conversion data of the first modality, wherein the dimension of the second conversion data is consistent with the dimension of the object data of the second modality; in response to the dimension of the first conversion data being consistent with the dimension of the object data of the second modality, using the first encoder to use the first conversion data as the target conversion data of the first modality.

[0210] Further, in a possible implementation of the embodiment of the present application, the conversion module 130 is also used to: in response to the dimension of the object data of the first modality being inconsistent with the dimension of the object data of the second modality, perform dimension conversion on the object data of the first modality through the first encoder to obtain third conversion data, wherein the dimension of the third conversion data is consistent with the dimension of the object data of the second modality; in response to the third coordinate system of the third conversion data being inconsistent with the second coordinate system of the object data of the second modality, perform coordinate conversion on the third conversion data through the first encoder to obtain fourth conversion data in the second coordinate system as target conversion data of the first modality; in response to the third coordinate system being consistent with the second coordinate system, use the first encoder to use the third conversion data as the target conversion data of the first modality.

[0211] Furthermore, in a possible implementation of an embodiment of the present application, the conversion module 130 is also used to: perform coordinate conversion on the first point cloud data through the first encoder to obtain second point cloud data in a camera coordinate system; perform dimension conversion on the second point cloud data through the first encoder to obtain third point cloud data as target conversion data of the point cloud modality, wherein the dimension of the third point cloud data is consistent with the dimension of the first image data.

[0212] Furthermore, in a possible implementation of the embodiment of the present application, the fusion module 140 is also used to: perform feature extraction on the first image data through the first encoder to obtain a first feature; perform feature extraction on the third point cloud data through the first encoder to obtain a second feature; and perform splicing of the first feature and the second feature through the first encoder to obtain the fused data.

[0213] Further, in a possible implementation manner of the embodiment of the present application, the fusion module 140 is further configured to: segment the first image data by using the first encoder to obtain second image data of each of N image blocks, where N is a positive integer; perform feature extraction on the second image data of the i-th image block by using the first encoder to obtain a first feature of the i-th image block, where i is a positive integer not greater than N;

[0214] The fusion module 140 is also used to: segment the third point cloud data through the first encoder to obtain fourth point cloud data of each of the N image blocks; and extract features of the fourth point cloud data of the i-th image block through the first encoder to obtain a second feature of the i-th image block.

[0215] Furthermore, in a possible implementation of an embodiment of the present application, the fusion module 140 is also used to: splice the first feature of the i-th image block and the second feature of the i-th image block through the first encoder to obtain fusion data of the i-th image block.

[0216] Furthermore, in a possible implementation of the embodiment of the present application, the large model also includes a feature extraction network, a decoder, a second encoder and a multilayer perceptron;

[0217] The detection module 150 is also used to: perform feature extraction on the fused data through the feature extraction network to obtain a third feature; perform target detection based on the third feature through the decoder to obtain a fourth feature; perform feature extraction on the third feature and the fused data through the second encoder to obtain a fifth feature; and obtain a segmentation mask corresponding to the first image data based on the fourth feature and the fifth feature through the multilayer perceptron.

[0218] Further, in a possible implementation of the embodiment of the present application, the feature extraction network includes a backbone network, a position encoder and a third encoder;

[0219] The detection module 150 is also used to: perform feature extraction on the fused data through the backbone network to obtain a sixth feature; perform position encoding on the sixth feature through the position encoder to obtain position encoding data of the sixth feature; perform feature extraction on the sixth feature and the position encoding data through the third encoder to obtain the third feature.

[0220] Furthermore, in a possible implementation manner of an embodiment of the present application, the detection module 150 is also used to: obtain a classification result corresponding to the first image data based on the fourth feature through the multilayer perceptron; and obtain a bounding box corresponding to the first image data based on the fourth feature through the multilayer perceptron.

[0221] Furthermore, in a possible implementation manner of the embodiment of the present application, the device 100 also includes: a training module, used to: obtain a first loss function of the large model based on the predicted classification result corresponding to the sample image data; obtain a second loss function of the large model based on the predicted bounding box corresponding to the sample image data; obtain a third loss function of the large model based on the predicted segmentation mask corresponding to the sample image data; obtain a predicted target detection result corresponding to the sample image data based on at least one of the predicted classification result, predicted bounding box and predicted segmentation mask corresponding to the sample image data; obtain a fourth loss function of the large model based on the predicted target detection result; train the large model based on at least two of the first loss function, the second loss function, the third loss function and the fourth loss function.

[0222] Furthermore, in a possible implementation method of an embodiment of the present application, the training module is also used to: obtain the fifth loss function of the large model based on the number of breakpoints contained in the predicted weld trajectory; obtain the sixth loss function of the large model based on the target distance between two adjacent breakpoints contained in the predicted weld trajectory; obtain the fourth loss function based on at least one of the fifth loss function and the sixth loss function.

[0223] Furthermore, in a possible implementation of an embodiment of the present application, the training module is also used to: obtain the sum of multiple target distances corresponding to the predicted weld trajectory as the sixth loss function.

[0224] It should be noted that the aforementioned explanation of the embodiment of the large model-based target detection method is also applicable to the large model-based target detection device of this embodiment, and will not be repeated here.

[0225] In summary, the target detection device based on a large model in the embodiment of the present application can use multimodal data for target detection, which improves the robustness, generalization and accuracy of target detection. The first encoder can be used to perform at least one of coordinate transformation and dimension transformation on the object data of a certain modality to obtain the target conversion data of the modality to achieve multimodal data fusion, and perform target detection based on the fused data through the remaining networks of the large model, which simplifies user operations, improves target detection efficiency, and optimizes the user experience in target detection scenarios, and is suitable for weld detection scenarios.

[0226] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the large model-based target detection method provided in the above embodiments.

[0227] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the large model-based target detection method provided by the above embodiments.

[0228] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the large model-based target detection method provided by the above embodiments.

[0229] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0230] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0231] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0232] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they contradict each other.

[0233] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood to control or imply relative importance or implicitly indicate the number of technical features controlled. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0234] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0235] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0236] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0237] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0238] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0239] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A target detection method based on a large model, characterized in that: include: Acquire multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object; Inputting the multimodal object data into a large model, wherein the large model comprises a first encoder; Performing at least one of coordinate conversion and dimension conversion on the object data of the first modality through the first encoder to obtain target conversion data of the first modality; fusing the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data; By using the remaining networks other than the first encoder in the large model, target detection is performed based on the fused data; wherein, The remaining networks except the first encoder in the large model include a feature extraction network, a decoder, a second encoder and a multilayer perceptron.

2. The method according to claim 1, characterized in that The performing at least one of coordinate conversion and dimension conversion on the object data of the first modality by the first encoder to obtain target conversion data of the first modality includes: In response to the first coordinate system of the object data of the first modality being inconsistent with the second coordinate system of the object data of the second modality, coordinate conversion is performed on the object data of the first modality by the first encoder to obtain first conversion data in the second coordinate system; In response to the inconsistency between the dimension of the first conversion data and the dimension of the object data of the second modality, performing dimension conversion on the first conversion data by the first encoder to obtain second conversion data as target conversion data of the first modality, wherein the dimension of the second conversion data is consistent with the dimension of the object data of the second modality; In response to the dimension of the first conversion data being consistent with the dimension of the object data of the second modality, the first conversion data is used as the target conversion data of the first modality by the first encoder.

3. The method according to claim 1, characterized in that The performing at least one of coordinate conversion and dimension conversion on the object data of the first modality by the first encoder to obtain target conversion data of the first modality includes: In response to the dimension of the object data of the first modality being inconsistent with the dimension of the object data of the second modality, performing dimension conversion on the object data of the first modality by the first encoder to obtain third converted data, wherein the dimension of the third converted data is consistent with the dimension of the object data of the second modality; In response to the third coordinate system of the third conversion data being inconsistent with the second coordinate system of the object data of the second modality, coordinate conversion is performed on the third conversion data by the first encoder to obtain fourth conversion data in the second coordinate system as target conversion data of the first modality; In response to the third coordinate system being consistent with the second coordinate system, the third conversion data is used as target conversion data of the first modality by the first encoder.

4. The method according to claim 1, characterized in that The performing at least one of coordinate conversion and dimension conversion on the object data of the first modality by the first encoder to obtain target conversion data of the first modality includes: Performing coordinate conversion on the first point cloud data by using the first encoder to obtain second point cloud data in a camera coordinate system; The second point cloud data is dimensionally converted by the first encoder to obtain third point cloud data as target conversion data of the point cloud modality, wherein the dimension of the third point cloud data is consistent with the dimension of the first image data.

5. The method according to claim 4, characterized in that The step of fusing the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data includes: Extracting features from the first image data by using the first encoder to obtain first features; Performing feature extraction on the third point cloud data by using the first encoder to obtain a second feature; The first feature and the second feature are concatenated by the first encoder to obtain the fused data.

6. The method according to claim 5, characterized in that The extracting features of the first image data by the first encoder to obtain a first feature includes: Segmenting the first image data by the first encoder to obtain second image data of N image blocks, where N is a positive integer; Performing feature extraction on the second image data of the i-th image block by the first encoder to obtain a first feature of the i-th image block, where i is a positive integer not greater than N; The extracting features of the third point cloud data by the first encoder to obtain a second feature includes: Segmenting the third point cloud data by the first encoder to obtain fourth point cloud data of respective N image blocks; The first encoder performs feature extraction on the fourth point cloud data of the i-th image block to obtain a second feature of the i-th image block.

7. The method according to claim 6, characterized in that The step of concatenating the first feature and the second feature by the first encoder to obtain the fused data includes: The first feature of the i-th image block and the second feature of the i-th image block are spliced ​​through the first encoder to obtain fused data of the i-th image block.

8. The method according to claim 1, characterized in that: The performing target detection based on the fused data through the remaining networks other than the first encoder in the large model includes: Performing feature extraction on the fused data through the feature extraction network to obtain a third feature; Performing target detection based on the third feature by the decoder to obtain a fourth feature; Performing feature extraction on the third feature and the fused data by the second encoder to obtain a fifth feature; A segmentation mask corresponding to the first image data is obtained by the multilayer perceptron based on the fourth feature and the fifth feature.

9. The method according to claim 8, characterized in that The feature extraction network includes a backbone network, a position encoder and a third encoder; The step of extracting features from the fused data through the feature extraction network to obtain a third feature includes: Performing feature extraction on the fused data through the backbone network to obtain a sixth feature; Position-encode the sixth feature by using the position encoder to obtain position encoding data of the sixth feature; The third feature is obtained by performing feature extraction on the sixth feature and the position encoding data through the third encoder.

10. The method according to claim 8, characterized in that The method further comprises: Obtaining a classification result corresponding to the first image data based on the fourth feature through the multilayer perceptron; A bounding box corresponding to the first image data is obtained based on the fourth feature through the multilayer perceptron.

11. The method according to claim 10, characterized in that The method further comprises: Based on the prediction classification results corresponding to the sample image data, a first loss function of the large model is obtained; Obtaining a second loss function of the large model based on a predicted bounding box corresponding to the sample image data; Obtaining a third loss function of the large model based on the predicted segmentation mask corresponding to the sample image data; Obtaining a predicted target detection result corresponding to the sample image data based on at least one of a predicted classification result, a predicted bounding box, and a predicted segmentation mask corresponding to the sample image data; Based on the predicted target detection result, obtaining a fourth loss function of the large model; The large model is trained based on at least two of the first loss function, the second loss function, the third loss function, and the fourth loss function.

12. The method according to claim 11, characterized in that The fourth loss function of the large model is obtained based on the predicted target detection result, including: Based on the number of fracture points included in the predicted weld trajectory, a fifth loss function of the large model is obtained; Based on the target distance between two adjacent fracture points included in the predicted weld trajectory, a sixth loss function of the large model is obtained; The fourth loss function is obtained based on at least one of the fifth loss function and the sixth loss function.

13. The method according to claim 12, characterized in that The sixth loss function of the large model is obtained based on the target distance between two adjacent breakpoints included in the predicted weld trajectory, including: The sum of multiple target distances corresponding to the predicted weld trajectory is obtained as the sixth loss function.

14. A target detection device based on a large model, characterized in that: include: An acquisition module, configured to acquire multimodal object data, wherein the multimodal object data includes first image data of the object and first point cloud data of the object; A processing module, configured to input the multimodal object data into a large model, wherein the large model comprises a first encoder; a conversion module, configured to perform at least one of coordinate conversion and dimension conversion on the object data of the first modality through the first encoder to obtain target conversion data of the first modality; a fusion module, configured to fuse the target conversion data of the first modality with the object data of the second modality through the first encoder to obtain fused data; A detection module, used to perform target detection based on the fused data through the remaining networks other than the first encoder in the large model; wherein, The remaining networks except the first encoder in the large model include a feature extraction network, a decoder, a second encoder and a multilayer perceptron.

15. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 13 when executed by a processor.

Images