Self-supervised learning methods, devices, computer equipment, and storage media for perceptual data

Abstract

This invention discloses a self-supervised learning method, apparatus, computer device, and storage medium for perceptual data. The method includes: acquiring data collected by a camera and data detected by millimeter-wave radar to obtain video data and millimeter-wave radar data; performing coordinate mapping on the millimeter-wave radar data to obtain a mapping result; inputting the video data into a detection model for target detection to obtain a detection result; confirming the detection result using the mapping result to obtain a confirmation result; determining whether the confirmation result indicates a missed or falsely detected target; if so, determining and labeling the information of the missed or falsely detected target to form training data; training the detection model using the training data; and updating the detection model when the average loss value of the detection model is less than a set loss value threshold. By implementing the method of this invention, self-supervised learning of the detection model can be achieved, thereby improving the efficiency of detection model iteration and increasing the detection rate of video targets.

Description

Technical Field

[0001] This invention relates to model iteration methods, and more specifically to self-supervised learning methods, apparatus, computer devices, and storage media for perceptual data. Background Technology

[0002] As autonomous driving transitions from single-vehicle intelligence to multi-agent intelligence involving vehicles and vehicle-to-infrastructure (V2I) collaboration, roadside edge holographic perception technology is becoming increasingly important. The fusion of millimeter-wave radar and camera data based on MEC (Multi-access / Mobile Edge Computing) is widely used, with artificial intelligence neural network video detection technology playing a crucial role. However, due to minor occlusion and environmental factors, detection models often experience low recognition rates, including missed detections and false detections of vehicles, pedestrians, and other targets.

[0003] Therefore, it is necessary to design a new method to achieve self-supervised learning of the detection model, so as to improve the efficiency of the detection model iteration and increase the detection rate of video targets. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, computer device and storage medium for self-supervised learning of perceptual data.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a self-supervised learning method for perceptual data, comprising:

[0006] Data is acquired from the camera and the millimeter-wave radar to obtain video data and millimeter-wave radar data.

[0007] The millimeter-wave radar data is then mapped to coordinates to obtain the mapping result;

[0008] The video data is input into the detection model for target detection to obtain the detection results;

[0009] The detection result is confirmed using the mapping result to obtain a confirmation result;

[0010] Determine whether the confirmation result indicates a target that has been missed or falsely detected;

[0011] If the confirmation result is that there are missed or falsely detected targets, then the information of the missed or falsely detected targets is determined and labeled to form training data;

[0012] The detection model is trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0013] The further technical solution is as follows: The coordinate mapping of the millimeter-wave radar data to obtain the mapping result includes:

[0014] Obtain the camera's internal and external parameters based on the camera's calibration;

[0015] The transformation matrix from world coordinate system to image coordinate system is obtained using the aforementioned intrinsic and extrinsic parameters;

[0016] The millimeter-wave radar data is transformed from the world coordinate system to the image coordinate system using a transformation matrix to obtain the mapping result.

[0017] The further technical solution is as follows: The step of using the mapping result to confirm the detection result to obtain a confirmation result includes:

[0018] The mapping result is then mapped onto the video data to obtain the processing result;

[0019] Determine whether the pixel position of the processing result is within the rectangular area corresponding to the detection result;

[0020] If the pixel position of the processing result is within the rectangular area corresponding to the detection result, then it is determined whether the target type of the processing result is consistent with the target type corresponding to the detection result;

[0021] If the target type of the processing result is consistent with the target type corresponding to the detection result, then the confirmation result is determined to be a target that has no missed or false detections.

[0022] If the pixel position of the processing result is not within the rectangular area corresponding to the detection result, then it is determined that the confirmation result has a target that was missed or falsely detected.

[0023] If the target type of the processing result is inconsistent with the target type corresponding to the detection result, it is determined that there are missed or falsely detected targets in the confirmation result.

[0024] The further technical solution is as follows: determining the information of the targets of the missed detections and false detections, and labeling it to form training data, includes:

[0025] For the missed targets, the pixel positions where the mapping results are located are segmented to obtain the rectangular region, type, and location information of the missed targets;

[0026] For falsely detected targets, determine the correct type, rectangular region, and location information, and combine them with the rectangular region, type, and location information of the missed targets to form the rectangular region, type, and location information of the missed and falsely detected targets.

[0027] The video images are labeled based on the rectangular regions, types, and locations of the targets that were missed or falsely detected, in order to form training data.

[0028] The further technical solution is as follows: The detection model is trained using the training data; when the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results, including:

[0029] When the accumulated amount of training data reaches a threshold, the detection model is automatically trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0030] The further technical solution is as follows: the detection model includes a deep learning-based target detection network.

[0031] The present invention also provides a self-supervised learning device for perceptual data, comprising:

[0032] The data acquisition unit is used to acquire data collected by the camera and data detected by the millimeter-wave radar to obtain video data and millimeter-wave radar data.

[0033] A coordinate mapping unit is used to perform coordinate mapping on the millimeter-wave radar data to obtain a mapping result;

[0034] The target detection unit is used to input the video data into the detection model to perform target detection and obtain the detection result;

[0035] A confirmation unit is used to confirm the detection result using the mapping result to obtain a confirmation result;

[0036] The judgment unit is used to determine whether the confirmation result is a target that has been missed or falsely detected.

[0037] The annotation unit is used to determine the information of the missed or falsely detected targets and annotate them if the confirmation result is that there are such targets, so as to form training data.

[0038] The training unit is used to train the detection model using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0039] A further technical solution is as follows: the coordinate mapping unit includes:

[0040] The parameter acquisition subunit is used to obtain the camera's intrinsic and extrinsic parameters based on the camera's calibration.

[0041] The matrix acquisition subunit is used to obtain the transformation matrix from the world coordinate system to the image coordinate system using the intrinsic and extrinsic parameters;

[0042] The transformation subunit is used to transform the millimeter-wave radar data from the world coordinate system to the image coordinate system according to the transformation matrix to obtain the mapping result.

[0043] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0044] The present invention also provides a storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0045] The beneficial effects of this invention compared with the prior art are as follows: This invention uses the radar-video fusion tracking technology of millimeter-wave radar and camera to align the original millimeter-wave radar data with the video detection data, and uses the millimeter-wave radar data to reconfirm the results of the video data detection, filter out the missed and false detection data in the detection results, form automatically labeled data, and automatically train the detection model to realize the self-supervised learning of the detection model, so as to improve the efficiency of the detection model iteration and improve the detection rate of video targets.

[0046] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram illustrating an application scenario of the self-supervised learning method for perceptual data provided in an embodiment of the present invention.

[0049] Figure 2 A flowchart illustrating the self-supervised learning method for perceptual data provided in an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of a sub-process of the self-supervised learning method for perceptual data provided in an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of a sub-process of the self-supervised learning method for perceptual data provided in an embodiment of the present invention;

[0052] Figure 5This is a schematic diagram of a sub-process of the self-supervised learning method for perceptual data provided in an embodiment of the present invention;

[0053] Figure 6 A schematic block diagram of a self-supervised learning device for perceptual data provided in an embodiment of the present invention;

[0054] Figure 7 A schematic block diagram of the coordinate mapping unit of the self-supervised learning device for perceptual data provided in an embodiment of the present invention;

[0055] Figure 8 A schematic block diagram of the verification unit of the self-supervised learning device for perceptual data provided in an embodiment of the present invention;

[0056] Figure 9 A schematic block diagram of the annotation unit of the self-supervised learning device for perceptual data provided in an embodiment of the present invention;

[0057] Figure 10 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0060] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0061] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0062] Please see Figure 1 and Figure 2 , Figure 1This is a schematic diagram illustrating an application scenario of the self-supervised learning method for perceptual data provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart illustrating the self-supervised learning method for perceptual data provided in this embodiment of the invention. This method is applied to a server. The server is an MEC device that interacts with remote servers, cameras, and millimeter-wave radar sensors. It uses data detected by the millimeter-wave radar to correct the detection results obtained from the video data input to the detection model, and automatically labels the video data to form new training data. After accumulating a certain amount, it automatically trains the detection model. The detection model is updated only when the average loss value meets the requirements. Using holographic perceptual data, self-supervised learning training of the deep learning target detection model is achieved. This perfectly solves the problems of high difficulty, tediousness, low efficiency, and unstable quality of manual labeling, as well as the significant impact of minor occlusion or environmental factors on video. Through continuous self-supervised learning and efficient automatic model iteration, the quality of holographic perceptual data is improved. High-quality holographic intersection perception information can be achieved, meeting the needs of large-scale deployment. The system can be built more quickly and flexibly, resulting in certain social and economic benefits.

[0063] Figure 2 This is a flowchart illustrating the self-supervised learning method for perceptual data provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S170.

[0064] S110: Acquire data collected by the camera and data detected by the millimeter-wave radar to obtain video data and millimeter-wave radar data.

[0065] In this embodiment, video data refers to data collected by the camera from the roadside edge of a designated area, carrying information about traffic participants within the intersection. This information includes the latitude and longitude location of the traffic participants, their type, and other relevant information. Millimeter-wave radar data refers to data containing information about traffic participants within the intersection detected by a millimeter-wave radar sensor.

[0066] Video data and millimeter-wave radar data constitute holographic perception data.

[0067] S120. Perform coordinate mapping on the millimeter-wave radar data to obtain the mapping result.

[0068] In this embodiment, the mapping result refers to the data formed after millimeter-wave radar data is mapped from the world coordinate system to the image coordinate system.

[0069] In one embodiment, please refer to Figure 3 The above step S120 may include steps S121 to S123.

[0070] S121. Obtain the camera's internal and external parameters based on the camera's calibration;

[0071] S122. Obtain the transformation matrix from world coordinate system to image coordinate system using the aforementioned intrinsic and extrinsic parameters;

[0072] S123. The millimeter-wave radar data is transformed from the world coordinate system to the image coordinate system according to the transformation matrix to obtain the mapping result.

[0073] In this embodiment, in order to align millimeter-wave radar data with video detection data in both space and time, the millimeter-wave radar device and camera need to be calibrated in space, and the NTP (Network Time Protocol) timing service needs to be activated in time to keep the clocks of the millimeter-wave radar and camera synchronized.

[0074] Specifically, coordinate mapping using the intrinsic and extrinsic parameters of a camera is existing technology and will not be elaborated upon here.

[0075] S130. Input the video data into the detection model to perform target detection and obtain the detection result.

[0076] In this embodiment, the detection result refers to the rectangular box formed by the type and location information of traffic participants in the video data.

[0077] In this embodiment, the detection model is obtained by collecting video data from cameras mounted on traffic enforcement poles, manually annotating the data to form a small amount of high-quality training data, and then training a basic deep learning object detection model. This basic deep learning object detection model is used to detect traffic participants, and the rectangular regions of traffic participants in the image coordinate system can be obtained based on the model. The camera type is, but is not limited to, a wide-angle fisheye camera, and the installation method is, but is not limited to, mounting directly above or to the side of the video monitoring area on the L-shaped traffic enforcement pole. The deep learning object detection model includes, but is not limited to, SequenceNet-SSD, MobileNetV2-SSD, Cascade-RCNN, and VGG-RepulsionLoss.

[0078] S140. The detection result is confirmed using the mapping result to obtain a confirmation result.

[0079] In this embodiment, the confirmation result refers to confirming whether there are any missed or falsely detected targets in the detection results.

[0080] In one embodiment, please refer to Figure 4 The above-mentioned step S140 may include steps S141 to S145.

[0081] S141. Map the mapping result onto the video data to obtain the processing result.

[0082] In this embodiment, the processing result refers to determining the position of the mapping result in the video data corresponding to the mapping result.

[0083] S142. Determine whether the pixel position of the processing result is within the rectangular area corresponding to the detection result;

[0084] S143. If the pixel position of the processing result is within the rectangular area corresponding to the detection result, then determine whether the target type of the processing result is consistent with the target type corresponding to the detection result.

[0085] S144. If the target type of the processing result is consistent with the target type corresponding to the detection result, then the confirmation result is determined to be a target that has no missed or false detections.

[0086] S145. If the pixel position of the processing result is not within the rectangular area corresponding to the detection result, it is determined that the confirmation result has a target that is missed or falsely detected.

[0087] If the target type of the processing result is inconsistent with the target type corresponding to the detection result, then step S145 is executed.

[0088] Specifically, when the pixel position of the processed result is within a rectangular area of ​​target data in the detection result, the millimeter-wave radar data and video data can be correlated. If the target type of the millimeter-wave radar data is inconsistent with the target type of the video data, it indicates that the video detection data is false detection data from the deep learning target detection model. When the pixel position of the processed result is not within the rectangular area of ​​the target data in the detection result, it indicates that the target data is missed detection data from the deep learning target detection model.

[0089] S150. Determine whether the confirmation result indicates a target that has been missed or falsely detected.

[0090] S160. If the confirmation result is that there are missed or falsely detected targets, then the information of the missed or falsely detected targets is determined and labeled to form training data.

[0091] In this embodiment, training data refers to the data formed by determining the rectangular region, type, and location information of the missed targets and annotating them in the video data.

[0092] In one embodiment, please refer to Figure 5 The above-mentioned step S160 may include steps S161 to S163.

[0093] S161. For the missed target, the pixel position where the mapping result is located is segmented to obtain the rectangular region, type and location information of the missed target.

[0094] S162. Determine the correct type, rectangular area, and location information for the falsely detected target, and combine them with the rectangular area, type, and location information of the missed target to form the rectangular area, type, and location information of the missed and falsely detected target.

[0095] S163. The video image is labeled according to the rectangular region, type, and location information of the targets that are missed or falsely detected, so as to form training data.

[0096] Specifically, when a missed detection occurs, it is necessary to segment the target near the pixel location of the millimeter-wave radar data. Through target segmentation, the rectangular region of the missed target, its location information, and its correct type information can be obtained.

[0097] In the case of false detections, the correct type, rectangular region, and location information are determined, and combined with the information of the missed targets to form new annotation content, which is then used to annotate the video data.

[0098] S170. The detection model is trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0099] Specifically, when the accumulated amount of training data reaches a threshold, the detection model is automatically trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0100] In this embodiment, once the training data accumulates to a certain amount, such as more than 5000 images, automatic iterative training of the model is initiated. The finally trained detection model undergoes a self-evaluation test. Only when the model's average loss (AVG Loss) value is less than 0.5 is the retrained detection model considered effective, and the previous detection model is updated. If the average loss value is greater than 0.5, no model update is performed, and the system waits for the next automatic iterative training to begin.

[0101] If the confirmation results do not show any missed or false detections, then proceed to the end step.

[0102] The method in this embodiment uses self-supervised learning based on roadside edge holographic perception data to iteratively train a deep learning detection model, improving the model's detection rate and providing high-quality holographic intersection perception information. It enables automatic and efficient model iteration under different scenarios and times, improving the detection rate of video targets and providing high-quality data support for roadside edge holographic perception data fusion.

[0103] Specifically, it enables self-supervised learning of video detection models through holographic perception data, achieving automatic and efficient model iteration and saving a significant amount of manpower costs associated with manual annotation; it also improves the detection rate of video targets, enhances the quality of holographic perception data, realizes a generalized road condition perception system, and provides high-quality data support for urban traffic management.

[0104] The aforementioned self-supervised learning method for perception data aligns the raw millimeter-wave radar data with video detection data using a radar-video fusion tracking technology that combines millimeter-wave radar and cameras. The millimeter-wave radar data is then used to reconfirm the results of the video data detection, filtering out missed and false detections and generating automatically labeled data. This data is then used to automatically train the detection model, achieving self-supervised learning of the detection model. This improves the efficiency of the detection model's iteration and increases the detection rate of video targets.

[0105] Figure 6 This is a schematic block diagram of a self-supervised learning device 300 for perceptual data provided in an embodiment of the present invention. Figure 6 As shown, corresponding to the above-described self-supervised learning method for perceptual data, the present invention also provides a self-supervised learning apparatus 300 for perceptual data. This self-supervised learning apparatus 300 includes a unit for executing the above-described self-supervised learning method for perceptual data, and the apparatus can be configured in a server. Specifically, please refer to... Figure 6 The self-supervised learning device 300 for perceptual data includes a data acquisition unit 301, a coordinate mapping unit 302, a target detection unit 303, a confirmation unit 304, a judgment unit 305, a labeling unit 306, and a training unit 307.

[0106] The data acquisition unit 301 is used to acquire data collected by the camera and data detected by the millimeter-wave radar to obtain video data and millimeter-wave radar data; the coordinate mapping unit 302 is used to perform coordinate mapping on the millimeter-wave radar data to obtain a mapping result; the target detection unit 303 is used to input the video data into the detection model for target detection to obtain a detection result; the confirmation unit 304 is used to confirm the detection result using the mapping result to obtain a confirmation result; the judgment unit 305 is used to determine whether the confirmation result indicates a target that has been missed or falsely detected; the annotation unit 306 is used to determine the information of the target that has been missed or falsely detected if the confirmation result indicates a target that has been missed or falsely detected, and to annotate it to form training data; the training unit 307 is used to train the detection model using the training data, and when the average loss value of the detection model is less than a set loss value threshold, the detection model is updated according to the training result.

[0107] In one embodiment, such as Figure 7 As shown, the coordinate mapping unit 302 includes a parameter acquisition subunit 3021, a matrix acquisition subunit 3022, and a transformation subunit 3023.

[0108] The parameter acquisition subunit 3021 is used to obtain the intrinsic and extrinsic parameters of the camera based on the camera calibration; the matrix acquisition subunit 3022 is used to obtain the transformation matrix from the world coordinate system to the image coordinate system using the intrinsic and extrinsic parameters; the transformation subunit 3023 is used to transform the millimeter-wave radar data from the world coordinate system to the image coordinate system according to the transformation matrix to obtain the mapping result.

[0109] In one embodiment, such as Figure 8 As shown, the confirmation unit 304 includes a processing subunit 3041, a position determination subunit 3042, a type determination subunit 3043, a first determination subunit 3044, and a second determination subunit 3045.

[0110] Processing subunit 3041 is used to map the mapping result to the video data to obtain a processing result; position determination subunit 3042 is used to determine whether the pixel position of the processing result is within the rectangular area corresponding to the detection result; type determination subunit 3043 is used to determine whether the target type of the processing result is consistent with the target type corresponding to the detection result if the pixel position of the processing result is within the rectangular area corresponding to the detection result; first determination subunit 3044 is used to determine that the confirmation result is a target without missed or false detection if the target type of the processing result is consistent with the target type corresponding to the detection result; second determination subunit 3045 is used to determine that the confirmation result has a target with missed or false detection if the pixel position of the processing result is not within the rectangular area corresponding to the detection result; and to determine that the confirmation result has a target with missed or false detection if the target type of the processing result is inconsistent with the target type corresponding to the detection result.

[0111] In one embodiment, such as Figure 9 As shown, the annotation unit 306 includes a first segmentation subunit 3061, a combination subunit 3062, and an automatic annotation subunit 3063.

[0112] The first segmentation subunit 3061 is used to segment the pixel position where the mapping result is located for the missed target to obtain the rectangular region, type and position information of the missed target; the combination subunit 3062 is used to determine the correct type, rectangular region and position information of the falsely detected target, and combine it with the rectangular region, type and position information of the missed target to form the rectangular region, type and position information of the missed and falsely detected target; the automatic annotation subunit 3063 is used to annotate the video image according to the rectangular region, type and position information of the missed and falsely detected target to form training data.

[0113] In one embodiment, the training unit 307 is used to automatically train the detection model using the training data when the accumulated amount of training data reaches a threshold, and to update the detection model based on the training results when the average loss value of the detection model is less than a set loss value threshold.

[0114] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned self-supervised learning device 300 for perceptual data and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0115] The aforementioned self-supervised learning device 300 for perceptual data can be implemented as a computer program, which can, for example... Figure 10It runs on the computer device shown.

[0116] Please see Figure 10 , Figure 10 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0117] See Figure 10 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0118] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a perceptual data self-supervised learning method.

[0119] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0120] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a perceptual data self-supervised learning method.

[0121] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0122] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0123] Data is acquired from a camera and detected by millimeter-wave radar to obtain video data and millimeter-wave radar data; coordinate mapping is performed on the millimeter-wave radar data to obtain a mapping result; the video data is input into a detection model for target detection to obtain a detection result; the mapping result is used to confirm the detection result to obtain a confirmation result; it is determined whether the confirmation result indicates a missed or falsely detected target; if the confirmation result indicates a missed or falsely detected target, the information of the missed or falsely detected target is determined and labeled to form training data; the detection model is trained using the training data, and when the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training result.

[0124] The detection model includes a deep learning-based object detection network.

[0125] In one embodiment, when the processor 502 performs the step of coordinate mapping on the millimeter-wave radar data to obtain the mapping result, it specifically implements the following steps:

[0126] The camera's intrinsic and extrinsic parameters are obtained based on the camera's calibration; the transformation matrix from the world coordinate system to the image coordinate system is obtained using the intrinsic and extrinsic parameters; the millimeter-wave radar data is transformed from the world coordinate system to the image coordinate system according to the transformation matrix to obtain the mapping result.

[0127] In one embodiment, when the processor 502 implements the step of confirming the detection result using the mapping result to obtain a confirmation result, the following steps are specifically implemented:

[0128] The mapping result is mapped onto the video data to obtain a processing result; it is determined whether the pixel position of the processing result is within the rectangular area corresponding to the detection result; if the pixel position of the processing result is within the rectangular area corresponding to the detection result, it is determined whether the target type of the processing result is consistent with the target type corresponding to the detection result; if the target type of the processing result is consistent with the target type corresponding to the detection result, it is determined that the confirmation result has no missed or false detection targets; if the pixel position of the processing result is not within the rectangular area corresponding to the detection result, it is determined that the confirmation result has missed or false detection targets; if the target type of the processing result is inconsistent with the target type corresponding to the detection result, it is determined that the confirmation result has missed or false detection targets.

[0129] In one embodiment, when the processor 502 implements the step of determining the information of the target of the missed detection and false detection and annotating it to form training data, it specifically implements the following steps:

[0130] For missed targets, the pixel positions where the mapping results are located are segmented to obtain the rectangular region, type, and location information of the missed targets; for falsely detected targets, the correct type, rectangular region, and location information are determined and combined with the rectangular region, type, and location information of the missed targets to form the rectangular region, type, and location information of the missed and falsely detected targets; the video images are labeled according to the rectangular region, type, and location information of the missed and falsely detected targets to form training data.

[0131] In one embodiment, when the processor 502 trains the detection model using the training data and updates the detection model based on the training results when the average loss value of the detection model is less than a set loss value threshold, the processor 502 specifically implements the following steps:

[0132] When the accumulated amount of training data reaches a threshold, the detection model is automatically trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0133] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0134] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0135] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps:

[0136] Data is acquired from a camera and detected by millimeter-wave radar to obtain video data and millimeter-wave radar data; coordinate mapping is performed on the millimeter-wave radar data to obtain a mapping result; the video data is input into a detection model for target detection to obtain a detection result; the mapping result is used to confirm the detection result to obtain a confirmation result; it is determined whether the confirmation result indicates a missed or falsely detected target; if the confirmation result indicates a missed or falsely detected target, the information of the missed or falsely detected target is determined and labeled to form training data; the detection model is trained using the training data, and when the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training result.

[0137] The detection model includes a deep learning-based object detection network.

[0138] In one embodiment, when the processor executes the computer program to perform coordinate mapping on the millimeter-wave radar data to obtain the mapping result, it specifically implements the following steps:

[0139] The camera's intrinsic and extrinsic parameters are obtained based on the camera's calibration; the transformation matrix from the world coordinate system to the image coordinate system is obtained using the intrinsic and extrinsic parameters; the millimeter-wave radar data is transformed from the world coordinate system to the image coordinate system according to the transformation matrix to obtain the mapping result.

[0140] In one embodiment, when the processor executes the computer program to implement the step of confirming the detection result using the mapping result to obtain a confirmation result, it specifically implements the following steps:

[0141] The mapping result is mapped onto the video data to obtain a processing result; it is determined whether the pixel position of the processing result is within the rectangular area corresponding to the detection result; if the pixel position of the processing result is within the rectangular area corresponding to the detection result, it is determined whether the target type of the processing result is consistent with the target type corresponding to the detection result; if the target type of the processing result is consistent with the target type corresponding to the detection result, it is determined that the confirmation result has no missed or false detection targets; if the pixel position of the processing result is not within the rectangular area corresponding to the detection result, it is determined that the confirmation result has missed or false detection targets; if the target type of the processing result is inconsistent with the target type corresponding to the detection result, it is determined that the confirmation result has missed or false detection targets.

[0142] In one embodiment, when the processor executes the computer program to determine the information of the target of the missed detection and false detection, and annotates it to form training data, it specifically implements the following steps:

[0143] For missed targets, the pixel positions where the mapping results are located are segmented to obtain the rectangular region, type, and location information of the missed targets; for falsely detected targets, the correct type, rectangular region, and location information are determined and combined with the rectangular region, type, and location information of the missed targets to form the rectangular region, type, and location information of the missed and falsely detected targets; the video images are labeled according to the rectangular region, type, and location information of the missed and falsely detected targets to form training data.

[0144] In one embodiment, when the processor executes the computer program to train the detection model using the training data, and updates the detection model based on the training results when the average loss value of the detection model is less than a set loss value threshold, the specific implementation of the following steps is as follows:

[0145] When the accumulated amount of training data reaches a threshold, the detection model is automatically trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

[0146] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0147] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0148] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0149] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0151] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A self-supervised learning method for perceptual data, characterized in that, include: Data is acquired from the camera and the millimeter-wave radar to obtain video data and millimeter-wave radar data. The millimeter-wave radar data is then mapped to coordinates to obtain the mapping result; The video data is input into the detection model for target detection to obtain the detection results; The detection result is confirmed using the mapping result to obtain a confirmation result; Determine whether the confirmation result indicates a target that has been missed or falsely detected; If the confirmation result is that there are missed or falsely detected targets, then the information of the missed or falsely detected targets is determined and labeled to form training data; The detection model is trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results. The process of determining and labeling the information of the targets of the missed and false detections to form training data includes: For the missed targets, the pixel positions where the mapping results are located are segmented to obtain the rectangular region, type, and location information of the missed targets; For falsely detected targets, determine the correct type, rectangular region, and location information, and combine them with the rectangular region, type, and location information of the missed targets to form the rectangular region, type, and location information of the missed and falsely detected targets. The video images are labeled based on the rectangular regions, types, and locations of the targets that were missed or falsely detected, in order to form training data.

2. The self-supervised learning method for perceptual data according to claim 1, characterized in that, The process of performing coordinate mapping on the millimeter-wave radar data to obtain the mapping result includes: Obtain the camera's internal and external parameters based on the camera's calibration; The transformation matrix from world coordinate system to image coordinate system is obtained using the aforementioned intrinsic and extrinsic parameters; The millimeter-wave radar data is transformed from the world coordinate system to the image coordinate system using a transformation matrix to obtain the mapping result.

3. The self-supervised learning method for perceptual data according to claim 1, characterized in that, The step of using the mapping result to confirm the detection result to obtain a confirmation result includes: The mapping result is then mapped onto the video data to obtain the processing result; Determine whether the pixel position of the processing result is within the rectangular area corresponding to the detection result; If the pixel position of the processing result is within the rectangular area corresponding to the detection result, then it is determined whether the target type of the processing result is consistent with the target type corresponding to the detection result; If the target type of the processing result is consistent with the target type corresponding to the detection result, then the confirmation result is determined to be a target that has no missed or false detections. If the pixel position of the processing result is not within the rectangular area corresponding to the detection result, then it is determined that the confirmation result has a target that was missed or falsely detected. If the target type of the processing result is inconsistent with the target type corresponding to the detection result, it is determined that there are missed or falsely detected targets in the confirmation result.

4. The self-supervised learning method for perceptual data according to claim 1, characterized in that, The step of training the detection model using the training data, and updating the detection model based on the training results when the average loss value of the detection model is less than a set loss value threshold, includes: When the accumulated amount of training data reaches a threshold, the detection model is automatically trained using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

5. The self-supervised learning method for perceptual data according to claim 1, characterized in that, The detection model includes a deep learning-based object detection network.

6. A self-supervised learning device for sensory data, characterized in that, include: The data acquisition unit is used to acquire data collected by the camera and data detected by the millimeter-wave radar to obtain video data and millimeter-wave radar data. A coordinate mapping unit is used to perform coordinate mapping on the millimeter-wave radar data to obtain a mapping result; The target detection unit is used to input the video data into the detection model to perform target detection and obtain the detection result; A confirmation unit is used to confirm the detection result using the mapping result to obtain a confirmation result; The judgment unit is used to determine whether the confirmation result is a target that has been missed or falsely detected. The annotation unit is used to determine the information of the missed or falsely detected targets and annotate them if the confirmation result is that there are such targets, so as to form training data. The annotation unit includes a first segmentation subunit, a combination subunit, and an automatic annotation subunit; The first segmentation subunit is used to segment the pixel position where the mapping result is located for the missed target, so as to obtain the rectangular region, type and location information of the missed target; The combination subunit is used to determine the correct type, rectangular area, and location information of the falsely detected target, and combine it with the rectangular area, type, and location information of the missed target to form the rectangular area, type, and location information of the falsely detected target. An automatic annotation subunit is used to annotate the video image based on the rectangular region, type, and location information of the missed and falsely detected targets to form training data; The training unit is used to train the detection model using the training data. When the average loss value of the detection model is less than a set loss value threshold, the detection model is updated based on the training results.

7. The self-supervised learning device for perceptual data according to claim 6, characterized in that, The coordinate mapping unit includes: The parameter acquisition subunit is used to obtain the camera's intrinsic and extrinsic parameters based on the camera's calibration. The matrix acquisition subunit is used to obtain the transformation matrix from the world coordinate system to the image coordinate system using the intrinsic and extrinsic parameters; The transformation subunit is used to transform the millimeter-wave radar data from the world coordinate system to the image coordinate system according to the transformation matrix to obtain the mapping result.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 5.

9. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images