Method, apparatus and device for detecting relative relationship of perception elements, and storage medium

Abstract

The application relates to the field of artificial intelligence, and discloses a sensing element relative relationship detection method, device, equipment and storage medium, the method comprising: acquiring a to-be-detected image photographed by a camera and first spatial information of multiple sensing elements detected by a sensor at the same time; inputting the to-be-detected image and the first spatial information of the multiple sensing elements into a preset spatial information adjustment model to obtain second spatial information of the multiple sensing elements; and calculating a relative relationship prediction value between the multiple sensing elements based on the second spatial information. The method designs a novel neural network architecture, which can directly capture the relative relationship between the sensing elements by using image information. Meanwhile, the relative relationship itself is directly modeled and supervised, so that the correctness of the relative relationship in the output result is ensured, and the instability problem of the relative relationship between the sensing elements caused by an endogenous position estimation error can be solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, device, and storage medium for detecting the relative relationships of sensing elements. Background Technology

[0002] With the development of artificial intelligence technology, autonomous driving has become a hot topic of attention and research. In vision-based autonomous driving perception systems, the inherent inaccuracy of 3D position estimation can cause errors between the relative position of the output object and the ground truth. For example, when depth estimation is performed separately for vehicles and lane lines in an image, the resulting 3D top-down view is prone to lateral position deviations. For instance, a vehicle may not be driving over the lane lines, but the detection result shows the vehicle's 3D bounding box is over the 3D lane lines. This error makes it difficult to determine whether a vehicle ahead intends to merge based on the relative relationship between the vehicle and the lane lines, affecting the stability of the autonomous driving system. Summary of the Invention

[0003] The main objective of this invention is to solve the technical problem of inaccurate relative relationships of objects caused by the inherent inaccuracy of 3D position estimation in existing autonomous driving perception systems.

[0004] The first aspect of the present invention provides a method for detecting the relative relationship of sensing elements, comprising: acquiring a camera-captured image to be detected and first spatial information of multiple sensing elements detected by a sensor at the same time; inputting the image to be detected and the first spatial information of the corresponding multiple sensing elements into a preset spatial information adjustment model to obtain second spatial information of the multiple sensing elements; and calculating a predicted value of the relative relationship between the multiple sensing elements based on the second spatial information.

[0005] Optionally, in a first implementation of the first aspect of the present invention, the step of inputting the first spatial information of the image to be detected and the corresponding plurality of perceptual elements into a preset spatial information adjustment model to obtain the second spatial information of the plurality of perceptual elements includes: inputting the first spatial information of the image to be detected and the corresponding plurality of perceptual elements into the preset spatial information adjustment model; extracting features from the image to be detected using the spatial information adjustment model to obtain image features of the image to be detected; projecting the corresponding perceptual elements onto the image to be detected using the spatial information adjustment model based on the first spatial information to obtain a set of pixel positions corresponding to the plurality of perceptual elements in the image to be detected; and adjusting the first spatial information of the perceptual elements based on the image features and the set of pixel positions to obtain the corresponding second spatial information.

[0006] Optionally, in a second implementation of the first aspect of the present invention, the spatial information adjustment model includes a feature extraction network, which includes a convolutional neural network and a feature pyramid network; the step of inputting the first spatial information of the image to be detected and the corresponding multiple perceptual elements into a preset spatial information adjustment model, and extracting features from the image to be detected through the spatial information adjustment model to obtain image features of the image to be detected includes: inputting the image to be detected into the preset spatial information adjustment model, performing convolution and pooling operations on the image to be detected through the convolutional neural network in the spatial information adjustment model to obtain feature maps of different depths; performing feature fusion on the feature maps through the feature pyramid network to obtain a fused feature map, and using the fused feature map as the image features of the image to be detected.

[0007] Optionally, in a third implementation of the first aspect of the present invention, the first spatial information is located on a preset world coordinate system; the step of projecting the corresponding sensing elements onto the image to be detected based on the first spatial information using the spatial information adjustment model to obtain the set of pixel positions corresponding to the plurality of sensing elements in the image to be detected includes: obtaining the intrinsic and extrinsic parameter information of the camera, and obtaining the intrinsic and extrinsic parameter matrices of the camera based on the intrinsic and extrinsic parameter information respectively; transforming the first spatial information of each sensing element from the world coordinate system to the camera coordinate system of the camera according to the intrinsic parameter matrix; converting the first spatial information on the camera coordinate system into pixel position coordinates on the image coordinate system of the image to be detected according to the extrinsic parameter matrix; and summarizing all pixel position coordinates corresponding to each sensing element to obtain the set of pixel positions corresponding to each sensing element in the image to be detected.

[0008] Optionally, in a fourth implementation of the first aspect of the present invention, adjusting the first spatial information of the perceptual element based on the image features and the pixel position set to obtain the corresponding second spatial information includes: calculating the image features of the corresponding perceptual element in the image to be detected based on the image features of the image to be detected and the pixel position set; fusing the image features of all the perceptual elements based on a preset self-attention mechanism to calculate the spatial offset of each perceptual element; and adjusting the corresponding first spatial information based on the spatial offset to obtain the corresponding second offset.

[0009] Optionally, in a fifth implementation of the first aspect of the present invention, the spatial information adjustment model is trained through the following steps: acquiring multiple historical detection images captured by a camera, third spatial information of multiple historical sensing elements detected by a sensor at the same time as the historical detection images, and the real spatial information of the historical sensing elements; projecting the corresponding historical sensing elements onto the historical detection images according to the third spatial information to obtain a set of pixel positions of multiple historical sensing elements on the historical detection images; extracting image features from the historical detection images according to a preset feature extraction network to obtain image features corresponding to the historical detection images; and calculating the image of the corresponding historical sensing elements based on the image features corresponding to the historical detection images and the set of pixel positions. The system performs image feature extraction; information fusion is performed on the image features of multiple historical sensing elements based on a self-attention mechanism to obtain the fourth spatial information of the historical sensing elements; a preset loss function is calculated based on the real spatial information and the fourth spatial information to obtain the loss function value; it is determined whether the loss function value is greater than a preset loss threshold; if so, the network parameters of the feature extraction network and the self-attention mechanism are adjusted according to the loss function value, and the loss function is repeatedly calculated based on the input historical detection images and historical sensing elements through the adjusted feature extraction network and the self-attention mechanism until the loss function value is less than or equal to the loss threshold; if not, a spatial information adjustment model is obtained based on the network parameters of the feature extraction network and the self-attention mechanism.

[0010] Optionally, in a sixth implementation of the first aspect of the present invention, the loss function includes a relative relationship loss function and / or a direct loss function; the step of calculating a preset loss function based on the real spatial information and the fourth spatial information to obtain the loss function value includes: calculating the true relative relationship values ​​of multiple historical sensing elements based on the real spatial information; calculating the predicted relative relationship values ​​of multiple historical sensing elements based on the fourth spatial information; calculating the direct loss function based on the real spatial information and the fourth spatial information, and / or calculating the relative relationship loss function based on the predicted relative relationship values ​​and the true relative relationship values ​​of multiple historical sensing elements.

[0011] A second aspect of the present invention provides a device for detecting the relative relationship of sensing elements, comprising: an acquisition module for acquiring a camera-captured image to be detected and first spatial information of multiple sensing elements detected by a sensor at the same time; an adjustment module for inputting the image to be detected and the first spatial information of the corresponding multiple sensing elements into a preset spatial information adjustment model to obtain second spatial information of the multiple sensing elements; and a calculation module for calculating a predicted value of the relative relationship between the multiple sensing elements based on the second spatial information.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the adjustment module specifically includes: a feature extraction unit, used to input the first spatial information of the image to be detected and the corresponding plurality of perceptual elements into a preset spatial information adjustment model, and to extract features from the image to be detected through the spatial information adjustment model to obtain image features of the image to be detected; a projection unit, used to project the corresponding perceptual elements onto the image to be detected according to the first spatial information through the spatial information adjustment model to obtain a set of pixel positions corresponding to the plurality of perceptual elements in the image to be detected; and a position adjustment unit, used to adjust the first spatial information of the perceptual elements based on the image features and the set of pixel positions to obtain corresponding second spatial information.

[0013] Optionally, in a second implementation of the second aspect of the present invention, the spatial information adjustment model includes a feature extraction network, which includes a convolutional neural network and a feature pyramid network; the feature extraction unit is specifically used to: input the image to be detected into a preset spatial information adjustment model, perform convolution and pooling operations on the image to be detected through the convolutional neural network in the spatial information adjustment model to obtain feature maps of different depths; perform feature fusion on the feature maps through the feature pyramid network to obtain a fused feature map, and use the fused feature map as the image feature of the image to be detected.

[0014] Optionally, in a third implementation of the second aspect of the present invention, the first spatial information is located on a preset world coordinate system; the projection unit is specifically used for: acquiring the intrinsic and extrinsic parameter information of the camera, and obtaining the intrinsic and extrinsic parameter matrices of the camera based on the intrinsic and extrinsic parameter information respectively; transforming the first spatial information of each sensing element from the world coordinate system to the camera coordinate system of the camera according to the intrinsic parameter matrix; converting the first spatial information on the camera coordinate system into pixel position coordinates on the image coordinate system of the image to be detected according to the extrinsic parameter matrix; summarizing all pixel position coordinates corresponding to each sensing element to obtain a set of pixel positions corresponding to each sensing element in the image to be detected.

[0015] Optionally, in a fourth implementation of the second aspect of the present invention, the position adjustment unit is specifically used to: calculate the image features of the corresponding perceptual element in the image to be detected based on the image features of the image to be detected and the set of pixel positions; fuse the image features of all the perceptual elements based on a preset self-attention mechanism to calculate the spatial offset of each perceptual element; and adjust the corresponding first spatial information based on the spatial offset to obtain the corresponding second offset.

[0016] Optionally, in a fifth implementation of the second aspect of the present invention, the sensing element relative relationship detection device further includes a model training module, which is specifically used for: acquiring multiple historical detection images captured by a camera, third spatial information of multiple historical sensing elements detected by a sensor at the same time when the historical detection images were captured, and real spatial information of the historical sensing elements; projecting the corresponding historical sensing elements onto the historical detection images according to the third spatial information to obtain a set of pixel positions of multiple historical sensing elements on the historical detection images; extracting image features from the historical detection images according to a preset feature extraction network to obtain image features corresponding to the historical detection images; and calculating the corresponding [image features] based on the image features corresponding to the historical detection images and the set of pixel positions. Image features of historical sensing elements; information fusion of image features of multiple historical sensing elements based on a self-attention mechanism to obtain the fourth spatial information of the historical sensing elements; calculation of a preset loss function based on the real spatial information and the fourth spatial information to obtain the loss function value; determination of whether the loss function value is greater than a preset loss threshold; if so, the network parameters of the feature extraction network and the self-attention mechanism are adjusted according to the loss function value, and the loss function is repeatedly calculated based on the input historical detection images and historical sensing elements through the adjusted feature extraction network and the self-attention mechanism until the loss function value is less than or equal to the loss threshold; if not, a spatial information adjustment model is obtained based on the network parameters of the feature extraction network and the self-attention mechanism.

[0017] Optionally, in a sixth implementation of the second aspect of the present invention, the loss function includes a relative relationship loss function and / or a direct loss function; the model training module is further configured to: calculate the true relative relationship values ​​of multiple historical sensing elements based on the real spatial information; calculate the predicted relative relationship values ​​of multiple historical sensing elements based on the fourth spatial information; calculate the direct loss function based on the real spatial information and the fourth spatial information, and / or calculate the relative relationship loss function based on the predicted relative relationship values ​​of multiple historical sensing elements and the true relative relationship values.

[0018] A third aspect of the present invention provides a sensor element relative relationship detection device, comprising: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a circuit; the at least one processor invokes the instructions in the memory to cause the sensor element relative relationship detection device to perform the steps of the sensor element relative relationship detection method described above.

[0019] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described method for detecting the relative relationships of perceptual elements.

[0020] In the technical solution of this invention, a first spatial information of a camera-captured image of the target element and multiple sensing elements detected by a sensor at the same time are acquired. The image of the target element and the corresponding first spatial information of the multiple sensing elements are input into a preset spatial information adjustment model to obtain second spatial information of the multiple sensing elements. Based on the second spatial information, a predicted value of the relative relationship between the multiple sensing elements is calculated. This method designs a novel neural network architecture that can directly capture the relative relationship between sensing elements using image information. Simultaneously, it directly supervises the modeling of the relative relationship itself, thereby ensuring the correctness of the relative relationship in the output result and solving the problem of instability in the relative relationship between sensing elements caused by endogenous position estimation errors. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the first embodiment of the relative relationship detection method of sensing elements in this invention;

[0022] Figure 2 This is a schematic diagram of a second embodiment of the method for detecting the relative relationship of sensing elements in this invention;

[0023] Figure 3 This is a schematic diagram of one embodiment of the relative relationship detection device for sensing elements in this invention;

[0024] Figure 4 This is a schematic diagram of another embodiment of the relative relationship detection device for sensing elements in this invention;

[0025] Figure 5 This is a schematic diagram of one embodiment of the relative relationship detection device for sensing elements in this invention. Detailed Implementation

[0026] This application provides a method, apparatus, device, and storage medium for detecting the relative relationships of perceived elements, which solves the technical problem of inaccurate relative relationships of objects caused by the inherent inaccuracy of 3D position estimation in existing autonomous driving perception systems.

[0027] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 The first embodiment of the method for detecting the relative relationship of sensing elements in this invention includes:

[0029] 101. Acquire the first spatial information of the image to be detected captured by the camera and the multiple perceptual elements detected by the sensor at the same time;

[0030] It is understood that the executing entity of this invention can be a sensing element relative relationship detection device, a terminal, or a server; no specific limitation is made here. This embodiment of the invention will be described using a server as an example.

[0031] In practical applications, camera technology is mature, stable, inexpensive, and rich in information, making cameras an important sensing element for autonomous driving perception. Compared with laser point clouds, camera images can provide richer details and texture information. The objects captured by the camera are mainly areas that need to be represented using point clouds, such as any area in an urban road environment that includes vehicles, pedestrians, traffic signs, or billboards. During autonomous driving, the area captured by the camera is the field of view area in front of the vehicle. In this embodiment, the result captured by the camera is the image to be detected.

[0032] In this embodiment, during the autonomous driving process, in addition to using cameras for detection, other sensors are also used to detect the surrounding environment of the autonomous vehicle. For example, LiDAR emits lasers to generate point cloud data in the surrounding environment. The upstream system uses the point cloud data to detect objects in the environment, which are the sensing elements, such as streetlights, traffic signs, lane lines, and vehicles in front. While determining the sensing elements, the autonomous driving perception system uses the point cloud data to estimate the 3D position of the sensing elements and obtain the first spatial information of the sensing elements.

[0033] 102. Input the first spatial information of the image to be detected and the corresponding multiple perceptual elements into a preset spatial information adjustment model to obtain the second spatial information of the multiple perceptual elements;

[0034] In practical applications, autonomous vehicles detect the relative relationships between the acquired sensing elements. However, due to the inherent inaccuracy of 3D position estimation in the autonomous driving perception system, errors may occur between the relative position of the output object and the true value. Therefore, it is necessary to eliminate the inaccuracy of the positional relationship of the sensing elements to improve the accuracy of the detection of the relative relationship of the sensing elements. In this embodiment, the first spatial information of the input sensing elements is adjusted by a preset spatial information adjustment model, and the second spatial information has a higher accuracy.

[0035] In this embodiment, the model training process of the spatial adjustment model is as follows: Multiple historical detection images captured by the camera, third spatial information of multiple historical sensing elements detected by the sensor at the same time in the historical detection images, and real spatial information of the historical sensing elements are acquired; the corresponding historical sensing elements are projected onto the historical detection images based on the third spatial information to obtain a set of pixel positions of multiple historical sensing elements on the historical detection images; image features are extracted from the historical detection images using a preset feature extraction network to obtain image features corresponding to the historical detection images; image features of the corresponding historical sensing elements are calculated based on the image features corresponding to the historical detection images and the set of pixel positions; and self-attention is used to further refine the model training process. The force mechanism fuses image features of multiple historical sensing elements to obtain fourth spatial information of the historical sensing elements; based on the real spatial information and the fourth spatial information, a preset loss function is calculated to obtain a loss function value; it is determined whether the loss function value is greater than a preset loss threshold; if so, the network parameters of the feature extraction network and the self-attention mechanism are adjusted according to the loss function value, and the adjusted feature extraction network and self-attention mechanism are used to repeatedly calculate the loss function based on the input historical detection images and historical sensing elements until the loss function value is less than or equal to the loss threshold; if not, a spatial information adjustment model is obtained based on the network parameters of the feature extraction network and the self-attention mechanism.

[0036] Specifically, the loss function includes a relative relationship loss function and / or a direct loss function. The direct loss function is calculated to ensure that the output of each sensing element remains close to the true value. The relative relationship loss function is calculated to ensure that the relative positional relationship of the output sensing elements approximates the true relative positional relationship. Based on the true spatial information and the fourth spatial information, a preset loss function is calculated. The loss function value is obtained mainly by calculating the true relative relationship values ​​of multiple historical sensing elements based on the true spatial information; calculating the predicted relative relationship values ​​of multiple historical sensing elements based on the fourth spatial information; calculating the direct loss function based on the true spatial information and the fourth spatial information; and / or calculating the relative relationship loss function based on the predicted relative relationship values ​​and the true relative relationship values ​​of multiple historical sensing elements.

[0037] In this embodiment, before calculating the relative relationship loss function, the different input sensing elements are judged to determine which relative relationships will be input into the relative relationship loss function. For example, the relative relationship between two sensing elements that are more than a preset distance will not be input into the relative relationship loss function for calculation.

[0038] 103. Based on the second spatial information, calculate the predicted relative relationship between multiple sensing elements.

[0039] In this embodiment, the second spatial information is the spatial position in the three-dimensional space generated by the autonomous driving perception system, which may be the coordinate position in space, etc. The distance between two perception elements, the relative position direction and the included angle between the two perception elements are calculated by the coordinate position of the perception elements, and the results obtained by the above calculation are used as the predicted value of the relative relationship between the two perception elements.

[0040] In this embodiment, the method acquires a camera-captured image of the target element and first spatial information of multiple sensing elements detected by a sensor at the same time. The image and the first spatial information of the corresponding sensing elements are then input into a preset spatial information adjustment model to obtain second spatial information of the sensing elements. Based on the second spatial information, a predicted value of the relative relationship between the sensing elements is calculated. This method designs a novel neural network architecture that can directly capture the relative relationship between sensing elements using image information. Simultaneously, it directly supervises the modeling of the relative relationship itself, thereby ensuring the correctness of the relative relationship in the output result and solving the instability problem of the relative relationship between sensing elements caused by endogenous position estimation errors.

[0041] Please see Figure 2 The second embodiment of the method for detecting the relative relationship of sensing elements in this invention includes:

[0042] 201. Acquire the first spatial information of the image to be detected captured by the camera and the multiple perceptual elements detected by the sensor at the same time;

[0043] 202. Input the image to be detected into the preset spatial information adjustment model. The convolutional neural network in the spatial information adjustment model performs convolution and pooling operations on the image to be detected to obtain feature maps of different depths.

[0044] 203. Feature maps are fused using a feature pyramid network to obtain a fused feature map, which is then used as the image feature of the image to be detected.

[0045] In this embodiment, a convolutional neural network (such as a residual neural network) and a feature pyramid network are used to extract multi-scale image features F, with a feature size of N x H x W x C, where N is the batch size, H is the height, W is the width, and C is the number of channels.

[0046] 204. Obtain the camera's intrinsic and extrinsic parameters, and based on the intrinsic and extrinsic parameters, obtain the camera's intrinsic and extrinsic parameter matrices respectively;

[0047] 205. Based on the intrinsic parameter matrix, transform the first spatial information of each sensing element from the world coordinate system to the camera coordinate system of the camera;

[0048] 206. Based on the extrinsic parameter matrix, convert the first spatial information in the camera coordinate system into pixel position coordinates in the image coordinate system of the image to be detected;

[0049] 207. Summarize the pixel position coordinates corresponding to each sensing element to obtain the set of pixel positions corresponding to each sensing element in the image to be detected;

[0050] In practical applications, each camera requires calibration. Camera calibration refers to converting a 3D point with [X, Y, Z] coordinates in the 3D world into a 2D pixel with [X, Y] coordinates. For camera calibration, it is necessary to calculate the transformation relationship from world coordinates to pixel coordinates using the camera coordinate system. In practical applications, the transformation from world coordinates to camera coordinates is called extrinsic parameter calibration, and the extrinsic parameters are called R (rotation matrix) and T (translation matrix). The transformation from camera coordinates to pixel coordinates is called intrinsic parameter calibration, which obtains the camera's internal parameters, such as focal length and optical center. For input sensing elements (such as obstacles, lane lines, etc.), based on their spatial position and the camera's intrinsic and extrinsic parameter information, the corresponding set of pixel positions on the image can be calculated.

[0051] 208. Calculate the image features of the corresponding perceptual elements in the image to be detected based on the image features and pixel location set of the image to be detected;

[0052] In this embodiment, for each sensing element, based on the image location information of the corresponding image, the image features corresponding to that sensing element can be obtained from the image. The size of the tensor P of the image features is N x K x C, where K represents the number of sensing elements and C represents the feature dimension.

[0053] 209. Based on the preset self-attention mechanism, the image features of all perceptual elements are fused to calculate the spatial offset of each perceptual element.

[0054] 210. Adjust the corresponding first spatial information based on the spatial offset to obtain the corresponding second offset;

[0055] In this embodiment, a self-attention mechanism is used for information fusion. This allows each perceptual element to acquire the features and correlation information of other perceptual elements. Finally, the offset of each perceptual element is output, resulting in the position output of the perceptual elements after network fine-tuning.

[0056] 211. Based on the second spatial information, calculate the predicted relative relationship between multiple sensing elements.

[0057] This embodiment, based on the previous embodiment, details the process of inputting the first spatial information of the image to be detected and the corresponding multiple perceptual elements into a preset spatial information adjustment model to obtain the second spatial information of the multiple perceptual elements. By inputting the first spatial information of the image to be detected and the corresponding multiple perceptual elements into the preset spatial information adjustment model, the spatial information adjustment model extracts features from the image to be detected, obtaining image features of the image to be detected. The spatial information adjustment model projects the corresponding perceptual elements onto the image to be detected based on the first spatial information, obtaining a set of pixel positions corresponding to the multiple perceptual elements in the image to be detected. Based on the image features and the set of pixel positions, the first spatial information of the perceptual elements is adjusted to obtain the corresponding second spatial information. This method designs a novel neural network architecture that can directly capture the relative relationships between perceptual elements using image information. Simultaneously, it directly models and supervises the relative relationships themselves, thereby ensuring the correctness of the relative relationships in the output results and solving the instability problem of the relative relationships between perceptual elements caused by endogenous position estimation errors.

[0058] The method for detecting the relative relationship of sensing elements in the embodiments of the present invention has been described above. The device for detecting the relative relationship of sensing elements in the embodiments of the present invention will be described below. Please refer to [link / reference]. Figure 3 One embodiment of the relative relationship detection device for sensing elements in this invention includes:

[0059] The acquisition module 301 is used to acquire the first spatial information of the image to be detected captured by the camera and the multiple sensing elements detected by the sensor at the same time.

[0060] The adjustment module 302 is used to input the image to be detected and the first spatial information of the corresponding multiple sensing elements into a preset spatial information adjustment model to obtain the second spatial information of the multiple sensing elements;

[0061] The calculation module 303 is used to calculate the predicted relative relationship between the multiple sensing elements based on the second spatial information.

[0062] In this embodiment of the invention, the sensor element relative relationship detection device operates the aforementioned sensor element relative relationship detection method. The device acquires a camera-captured image of the target element and first spatial information of multiple sensor elements detected by a sensor at the same time. It then inputs the image of the target element and the corresponding first spatial information of the multiple sensor elements into a preset spatial information adjustment model to obtain second spatial information of the multiple sensor elements. Based on the second spatial information, it calculates predicted values ​​of the relative relationships between the multiple sensor elements. This method designs a novel neural network architecture that can directly capture the relative relationships between sensor elements using image information. Simultaneously, it directly supervises the modeling of the relative relationships themselves, thereby ensuring the correctness of the relative relationships in the output results and solving the instability problem of the relative relationships between sensor elements caused by endogenous position estimation errors.

[0063] Please see Figure 4 A second embodiment of the relative relationship detection device for sensing elements in this invention includes:

[0064] The acquisition module 301 is used to acquire the first spatial information of the image to be detected captured by the camera and the multiple sensing elements detected by the sensor at the same time.

[0065] The adjustment module 302 is used to input the image to be detected and the first spatial information of the corresponding multiple sensing elements into a preset spatial information adjustment model to obtain the second spatial information of the multiple sensing elements;

[0066] The calculation module 303 is used to calculate the predicted relative relationship between the multiple sensing elements based on the second spatial information.

[0067] In this embodiment, the adjustment module 302 specifically includes: a feature extraction unit 3021, used to input the first spatial information of the image to be detected and the corresponding multiple perceptual elements into a preset spatial information adjustment model, and to extract features from the image to be detected through the spatial information adjustment model to obtain the image features of the image to be detected; a projection unit 3022, used to project the corresponding perceptual elements onto the image to be detected according to the first spatial information through the spatial information adjustment model to obtain the set of pixel positions corresponding to the multiple perceptual elements in the image to be detected; and a position adjustment unit 3023, used to adjust the first spatial information of the perceptual elements based on the image features and the set of pixel positions to obtain the corresponding second spatial information.

[0068] In this embodiment, the spatial information adjustment model includes a feature extraction network, which includes a convolutional neural network and a feature pyramid network. The feature extraction unit 3021 is specifically used to: input the image to be detected into a preset spatial information adjustment model; perform convolution and pooling operations on the image to be detected through the convolutional neural network in the spatial information adjustment model to obtain feature maps of different depths; perform feature fusion on the feature maps through the feature pyramid network to obtain a fused feature map; and use the fused feature map as the image feature of the image to be detected.

[0069] In this embodiment, the first spatial information is located on a preset world coordinate system; the projection unit 3022 is specifically used for: acquiring the intrinsic and extrinsic parameter information of the camera, and obtaining the intrinsic and extrinsic parameter matrices of the camera based on the intrinsic and extrinsic parameter information respectively; transforming the first spatial information of each sensing element from the world coordinate system to the camera coordinate system of the camera according to the intrinsic parameter matrix; converting the first spatial information on the camera coordinate system into pixel position coordinates on the image coordinate system of the image to be detected according to the extrinsic parameter matrix; summarizing all pixel position coordinates corresponding to each sensing element to obtain the set of pixel positions corresponding to each sensing element in the image to be detected.

[0070] In this embodiment, the position adjustment unit 3023 is specifically used to: calculate the image features of the corresponding perceptual element in the image to be detected based on the image features of the image to be detected and the set of pixel positions; fuse the image features of all the perceptual elements based on a preset self-attention mechanism to calculate the spatial offset of each perceptual element; and adjust the corresponding first spatial information based on the spatial offset to obtain the corresponding second offset.

[0071] In this embodiment, the relative relationship detection device for sensing elements further includes a model training module 304, which is specifically used for: acquiring multiple historical detection images captured by a camera, third spatial information of multiple historical sensing elements detected by a sensor at the same time when the historical detection images were captured, and real spatial information of the historical sensing elements; projecting the corresponding historical sensing elements onto the historical detection images according to the third spatial information to obtain a set of pixel positions of multiple historical sensing elements on the historical detection images; extracting image features from the historical detection images according to a preset feature extraction network to obtain image features corresponding to the historical detection images; and calculating the corresponding historical sensing elements based on the image features corresponding to the historical detection images and the set of pixel positions. The image features of the elements are analyzed; information fusion is performed on the image features of multiple historical sensing elements based on a self-attention mechanism to obtain the fourth spatial information of the historical sensing elements; a preset loss function is calculated based on the real spatial information and the fourth spatial information to obtain the loss function value; it is determined whether the loss function value is greater than a preset loss threshold; if so, the network parameters of the feature extraction network and the self-attention mechanism are adjusted according to the loss function value, and the loss function is repeatedly calculated based on the input historical detection images and historical sensing elements through the adjusted feature extraction network and the self-attention mechanism until the loss function value is less than or equal to the loss threshold; if not, a spatial information adjustment model is obtained based on the network parameters of the feature extraction network and the self-attention mechanism.

[0072] In this embodiment, the loss function includes a relative relationship loss function and / or a direct loss function; the model training module 304 is further configured to: calculate the true relative relationship values ​​of multiple historical sensing elements based on the real spatial information; calculate the predicted relative relationship values ​​of multiple historical sensing elements based on the fourth spatial information; calculate the direct loss function based on the real spatial information and the fourth spatial information, and / or calculate the relative relationship loss function based on the predicted relative relationship values ​​of multiple historical sensing elements and the true relative relationship values.

[0073] This implementation details the specific functions of each module and the unit composition of some modules in the relative relationship detection device for sensing elements. Through each module and unit of this device, a model is pre-trained using historical detection images and historical sensing elements. This model, through a novel neural network architecture, can directly capture the relative relationships between sensing elements using image information. Simultaneously, it directly supervises the modeling of the relative relationships themselves, thereby ensuring the correctness of the relative relationships in the output results. This solves the problem of instability in the relative relationships between sensing elements caused by endogenous position estimation errors.

[0074] above Figure 3 and Figure 4 The relative relationship detection device of sensing elements in the embodiments of the present invention will be described in detail from the perspective of modular functional entities. The relative relationship detection device of sensing elements in the embodiments of the present invention will be described in detail from the perspective of hardware processing.

[0075] Figure 5 This is a schematic diagram of the structure of a sensing element relative relationship detection device 500 provided in an embodiment of the present invention. The sensing element relative relationship detection device 500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 510 (e.g., one or more processors) and a memory 520, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 533 or data 532. The memory 520 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the sensing element relative relationship detection device 500. Furthermore, the processor 510 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the sensing element relative relationship detection device 500 to implement the steps of the aforementioned sensing element relative relationship detection method.

[0076] The relative relationship detection device 500 may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input / output interfaces 560, and / or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 5 The illustrated structure of the sensing element relative relationship detection device does not constitute a limitation on the sensing element relative relationship detection device provided in this application. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0077] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the method for detecting the relative relationship of the sensing elements.

[0078] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0079] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0080] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting the relative relationship of sensing elements, characterized in that, The method for detecting the relative relationship of sensing elements includes: Acquire the first spatial information of the image to be detected captured by the camera and the multiple perceptual elements detected by the sensor at the same time; The image to be detected and the first spatial information of the corresponding multiple sensing elements are input into a preset spatial information adjustment model to obtain the second spatial information of the multiple sensing elements. Based on the second spatial information, a predicted value of the relative relationship between multiple sensing elements is calculated; the predicted value of the relative relationship includes the distance, relative position direction, and included angle between two sensing elements; The spatial information adjustment model is trained through the following steps: Acquire multiple historical detection images captured by the camera, third spatial information of multiple historical sensing elements detected by the sensor at the same moment when the historical detection images were captured, and real spatial information of the historical sensing elements; Based on the third spatial information, the corresponding historical sensing elements are projected onto the historical detection image to obtain a set of pixel positions of multiple historical sensing elements on the historical detection image; Image features are extracted from the historical detection images according to a preset feature extraction network to obtain the image features corresponding to the historical detection images; Based on the image features corresponding to the historical detection images and the set of pixel positions, calculate the image features of the corresponding historical sensing elements; Information fusion is performed on the image features of multiple historical sensing elements based on a self-attention mechanism to obtain the fourth spatial information of the historical sensing elements; Based on the real spatial information and the fourth spatial information, a preset loss function is calculated to obtain the loss function value; Determine whether the loss function value is greater than a preset loss threshold; If so, the network parameters of the feature extraction network and the self-attention mechanism are adjusted according to the loss function value, and the loss function is repeatedly calculated based on the input historical detection images and historical perception elements through the adjusted feature extraction network and self-attention mechanism until the loss function value is less than or equal to the loss threshold. If not, then the spatial information adjustment model is obtained based on the network parameters of the feature extraction network and the self-attention mechanism.

2. The method for detecting the relative relationship of sensing elements according to claim 1, characterized in that, The step of inputting the image to be detected and the first spatial information of the corresponding plurality of perceptual elements into a preset spatial information adjustment model to obtain the second spatial information of the plurality of perceptual elements includes: The first spatial information of the image to be detected and the corresponding multiple perceptual elements is input into a preset spatial information adjustment model. The spatial information adjustment model is used to extract features from the image to be detected to obtain the image features of the image to be detected. The spatial information adjustment model projects the corresponding perceptual elements onto the image to be detected based on the first spatial information, thereby obtaining a set of pixel positions corresponding to the plurality of perceptual elements in the image to be detected. Based on the image features and the set of pixel positions, the first spatial information of the perceptual element is adjusted to obtain the corresponding second spatial information.

3. The method for detecting the relative relationship of sensing elements according to claim 2, characterized in that, The spatial information adjustment model includes a feature extraction network, which includes a convolutional neural network and a feature pyramid network. The step of inputting the first spatial information of the image to be detected and the corresponding multiple perceptual elements into a preset spatial information adjustment model, and extracting features from the image to be detected through the spatial information adjustment model to obtain the image features of the image to be detected includes: The image to be detected is input into a preset spatial information adjustment model. The convolutional neural network in the spatial information adjustment model performs convolution and pooling operations on the image to be detected to obtain feature maps of different depths. The feature map is fused using the feature pyramid network to obtain a fused feature map, which is then used as the image feature of the image to be detected.

4. The method for detecting the relative relationship of sensing elements according to claim 2, characterized in that, The first spatial information is located in a preset world coordinate system; The step of adjusting the spatial information model to project the corresponding perceptual elements onto the image to be detected based on the first spatial information, and obtaining the set of pixel positions corresponding to the plurality of perceptual elements in the image to be detected, includes: Obtain the intrinsic and extrinsic parameters of the camera, and obtain the intrinsic and extrinsic parameter matrices of the camera based on the intrinsic and extrinsic parameters, respectively. Based on the intrinsic parameter matrix, the first spatial information of each sensing element is transformed from the world coordinate system to the camera coordinate system of the camera; Based on the extrinsic parameter matrix, the first spatial information in the camera coordinate system is converted into pixel position coordinates in the image coordinate system of the image to be detected; The pixel position coordinates corresponding to each sensing element are summarized to obtain the set of pixel positions corresponding to each sensing element in the image to be detected.

5. The method for detecting the relative relationship of sensing elements according to claim 2, characterized in that, The step of adjusting the first spatial information of the perceptual element based on the image features and the pixel location set to obtain the corresponding second spatial information includes: Based on the image features of the image to be detected and the set of pixel positions, calculate the image features of the corresponding perceptual elements in the image to be detected; Based on a preset self-attention mechanism, the image features of all the sensory elements are fused to calculate the spatial offset of each sensory element. The corresponding first spatial information is adjusted based on the spatial offset to obtain the corresponding second offset.

6. The method for detecting the relative relationship of sensing elements according to claim 1, characterized in that, The loss function includes a relative relationship loss function and / or a direct loss function; The step of calculating a preset loss function based on the real spatial information and the fourth spatial information, and obtaining the loss function value, includes: Based on the real spatial information, calculate the truth values ​​of the relative relationships of multiple historical sensing elements; Based on the fourth spatial information, calculate the predicted relative relationships of multiple historical sensing elements; The direct loss function is calculated based on the real spatial information and the fourth spatial information, and / or the relative relationship loss function is calculated based on the predicted relative relationship values ​​of multiple historical sensing elements and the true relative relationship values.

7. A device for detecting the relative relationship of sensing elements, characterized in that, The relative relationship detection device for sensing elements includes: The acquisition module is used to acquire the first spatial information of the image to be detected captured by the camera and the multiple perceptual elements detected by the sensor at the same time. The adjustment module is used to input the image to be detected and the first spatial information of the corresponding multiple sensing elements into a preset spatial information adjustment model to obtain the second spatial information of the multiple sensing elements. The calculation module is used to calculate the predicted relative relationship between multiple sensing elements based on the second spatial information; the predicted relative relationship includes the distance, relative position direction and included angle between two sensing elements; The spatial information adjustment model is trained through the following steps: acquiring multiple historical detection images captured by a camera, third spatial information of multiple historical sensing elements detected by a sensor at the same time when the historical detection images were captured, and the real spatial information of the historical sensing elements; projecting the corresponding historical sensing elements onto the historical detection images based on the third spatial information to obtain a set of pixel positions of multiple historical sensing elements on the historical detection images; extracting image features from the historical detection images using a preset feature extraction network to obtain image features corresponding to the historical detection images; calculating image features of the corresponding historical sensing elements based on the image features corresponding to the historical detection images and the set of pixel positions; and using an autoattention mechanism... The system fuses image features of multiple historical sensing elements to obtain fourth spatial information of the historical sensing elements; based on the real spatial information and the fourth spatial information, a preset loss function is calculated to obtain a loss function value; it is determined whether the loss function value is greater than a preset loss threshold; if so, the network parameters of the feature extraction network and the self-attention mechanism are adjusted according to the loss function value, and the adjusted feature extraction network and self-attention mechanism are used to repeatedly calculate the loss function based on the input historical detection images and historical sensing elements until the loss function value is less than or equal to the loss threshold; if not, a spatial information adjustment model is obtained based on the network parameters of the feature extraction network and the self-attention mechanism.

8. A device for detecting the relative relationship of sensing elements, characterized in that, The sensing element relative relationship detection device includes: a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected via a line; The at least one processor invokes the instructions in the memory to cause the sensing element relative relationship detection device to perform the steps of the sensing element relative relationship detection method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting the relative relationship of perceptual elements as described in any one of claims 1-6.

Images