Training methods and apparatus for object detection models
By excluding the loss function of the current frame image during the training of the target detection model and combining it with the extraction of image features at equal intervals, the problem of the inability to effectively utilize the temporal information of continuous frames in the existing technology is solved, thereby improving the 3D target recognition performance of the model.
Patent Information
- Application Number
- CN202410150533.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-02-02
AI Technical Summary
In existing pure vision-based 3D object detection technologies, the temporal and motion information of consecutive frames cannot be effectively utilized during model training, making it difficult to improve model performance.
By excluding the loss function construction process of the current frame image, the three-dimensional spatial features of images in consecutive frames other than the current frame are stitched together into specific fusion features. Combined with image features extracted at multiple equal intervals, multiple loss functions are formed to train the target detection model.
It improves the accuracy and capability of the model in 3D target recognition, effectively utilizes the rich temporal information of historical frames, and avoids the defect of the model over-relying on the current frame.
Smart Images

Figure CN117994752B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and apparatus for training an object detection model. Background Technology
[0002] In the field of autonomous driving, real-time identification of the 3D information of targets is required for scene perception, control, and planning. Current pure vision-based 3D target detection technologies typically use the current frame for model training or incorporate a certain number of historical frames for auxiliary training. The former method fails to utilize the temporal motion features of consecutive frames, while the latter overemphasizes the information of the current frame and neglects historical frame information. This is because the labeled ground truth values in the training data correspond to the current frame, naturally leading to a bias in model training. Consequently, the model struggles to learn the rich temporal and motion information from historical frames, hindering further performance improvements. Summary of the Invention
[0003] In view of this, embodiments of the present invention provide a training method and apparatus for an object detection model, which improves model performance by excluding the loss function construction process of the current frame image during model training.
[0004] To achieve the above objectives, according to one aspect of the present invention, a method for training an object detection model is provided.
[0005] The training method for the target detection model in this embodiment of the invention includes: acquiring video information as input data for model training, and extracting three-dimensional spatial features of consecutive frames in the video information; wherein the consecutive frames include the current frame as the last frame; concatenating the three-dimensional spatial features of other images in the consecutive frames except the current frame into a specific fusion feature and inputting it into the target detection model to be trained; obtaining a second type of loss function based on the output result of the target detection model and the labeled ground truth data of the current frame; forming a final loss function based on the second type of loss function, and training the target detection model using the final loss function.
[0006] Optionally, forming the final loss function based on the second type of loss function includes: determining the second type of loss function as the final loss function.
[0007] Optionally, the method further includes: after extracting the three-dimensional spatial features of consecutive frames in the video information, performing multiple extractions in the consecutive frames, and determining multiple frames of images with equally spaced sequence numbers in the consecutive frames based on each extraction; wherein the sequence number intervals of different extractions are different, and the sequence number interval is not less than 2; concatenating the three-dimensional spatial features of the images determined in each extraction into the temporal fusion features of that extraction and inputting them into the target detection model, obtaining the first type of loss function for that extraction based on the output result of the target detection model and the labeled ground truth data of the current frame; and forming the final loss function based on the second type of loss function includes: combining the first type of loss function and the second type of loss function of the multiple extractions into the final loss function.
[0008] Optionally, extracting the three-dimensional spatial features of consecutive frames in the video information includes: obtaining feature map features of each frame in the consecutive frames using a preset feature extraction model; and performing bird's-eye view projection processing on the feature map features to obtain the three-dimensional spatial features.
[0009] Optionally, determining the multiple frames of images arranged at equal intervals in sequence number in the consecutive frames according to each extraction includes: in each extraction, taking the current frame as the first frame image determined in this extraction, and extracting images sequentially in descending order of time from the current frame according to the sequence number interval of this extraction.
[0010] Optionally, the sequence number interval of the multiple extractions covers an integer from 2 to a first value, where the first value is half the number of images in the consecutive frames.
[0011] Optionally, the method further includes: concatenating the three-dimensional spatial features of each image in the consecutive frames into a full fusion feature and inputting it into the target detection model; obtaining a third type of loss function based on the output of the target detection model and the labeled ground truth data of the current frame; and combining the multiple extracted first type of loss function and second type of loss function into a final loss function, including: adding the multiple extracted first type of loss function, second type of loss function and third type of loss function to obtain the final loss function.
[0012] Optionally, the labeled ground truth data includes: target type, and also includes at least one of the following: target position, target pose, and target size.
[0013] To achieve the above objectives, according to another aspect of the present invention, a target detection method is provided.
[0014] The target detection method of this invention includes: extracting the three-dimensional spatial features of each frame image in the video information where the last frame is the frame to be detected; stitching the three-dimensional spatial features of each frame image together and inputting them into the trained target detection model to obtain the detection result in the frame to be detected; wherein, the target detection model is trained according to the above target detection model training method.
[0015] Optionally, the detection result includes: target type, and also includes at least one of the following data: target position, target pose, and target size.
[0016] To achieve the above objectives, according to another aspect of the present invention, a training apparatus for an object detection model is provided.
[0017] The training apparatus for the target detection model in this embodiment of the invention may include: a feature extraction unit, configured to acquire video information as input data for model training, and extract three-dimensional spatial features of consecutive frames in the video information; wherein the consecutive frames include the current frame as the last frame; a second function construction unit, configured to concatenate the three-dimensional spatial features of other images in the consecutive frames except the current frame into specific fusion features and input them into the target detection model to be trained, and obtain a second type of loss function based on the output result of the target detection model and the labeled ground truth data of the current frame; and a training unit, configured to form a final loss function based on the second type of loss function, and train the target detection model using the final loss function.
[0018] To achieve the above objectives, according to another aspect of the present invention, a target detection device is provided.
[0019] The target detection device of this invention can be used to: extract the three-dimensional spatial features of each frame image in video information where the last frame is the frame to be detected, and input the three-dimensional spatial features of each frame image into the trained target detection model to obtain the detection result in the frame to be detected; wherein, the target detection model is trained based on the above target detection model training device.
[0020] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.
[0021] An electronic device according to the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method or object detection method of the object detection model provided by the present invention.
[0022] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided.
[0023] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method or a target detection method for the target detection model provided by the present invention.
[0024] According to the technical solution of the present invention, the embodiments described above have the following advantages or beneficial effects:
[0025] To avoid the drawback of excessive favoring of the current frame and neglect of historical frames caused by the existing feature fusion of the current frame and historical frames, this invention adopts the following approach: the three-dimensional spatial features of historical frames other than the current frame in consecutive frames are spliced together, and a second type of loss function is constructed based on the spliced features to form the final loss function for training the target detection model. This approach avoids the defect of the model relying too much on the current frame by abandoning the traditional feature fusion method of the current frame and historical frames, which helps to improve the accuracy of three-dimensional target recognition. More preferably, this embodiment of the invention combines the above method with another feature fusion method, namely, multiple extractions are performed in a continuous frame including the current frame and historical frames. Based on each extraction, multiple frames of images with equally spaced sequence numbers are determined in the continuous frame, wherein the sequence number interval of each extraction is different and not less than 2. Then, the three-dimensional spatial features of the images determined in each extraction are stitched together and a first type of loss function is constructed based on the stitched features. Finally, the first type of loss function and the second type of loss function are combined to generate the final loss function to train the target detection model. This combination scheme, by abandoning the traditional feature fusion method of current frame and historical frame and retaining the rich temporal motion information in the current frame and historical frame, more effectively avoids the defect of the model over-reliance on the current frame, which helps to further improve the three-dimensional target recognition capability of the model.
[0026] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0027] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:
[0028] Figure 1 This is a schematic diagram illustrating the main steps of the training method for the target detection model in this embodiment of the invention;
[0029] Figure 2 This is a schematic diagram of the model structure according to an embodiment of the present invention;
[0030] Figure 3 This is a schematic diagram of the components of the training device for the target detection model in an embodiment of the present invention;
[0031] Figure 4 This is an exemplary system architecture diagram that can be applied thereto according to embodiments of the present invention;
[0032] Figure 5 This is a schematic diagram of the electronic device structure used to implement the training method of the target detection model in the embodiments of the present invention. Detailed Implementation
[0033] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0034] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.
[0035] Figure 1 This is a schematic diagram illustrating the main steps of the training method for the target detection model according to an embodiment of the present invention.
[0036] like Figure 1 As shown, the training method of the target detection model in this embodiment of the invention can be specifically performed according to the following steps:
[0037] Step S101: Obtain video information as input data for model training, and extract the three-dimensional spatial features of consecutive frames in the video information.
[0038] In this step, the server used to perform model training can first obtain multiple pre-annotated video information segments as input data for model training. For example, in a 3D object detection scenario, the ground truth data of each video information segment can be the ground truth of the last frame (i.e., the current frame) in the continuous frames contained in the video information. The ground truth data can include one or more of the following: target type (e.g., whether the target is a person or a vehicle), target position (including the target's 3D coordinates in real space), target pose (e.g., the target's azimuth angle, pitch angle, etc.), and target size (e.g., the target's length, width, and height).
[0039] In practical applications, after acquiring video information, the server can first use a pre-defined feature extraction model (such as a convolutional neural network) to obtain the feature map features of each frame in a series of frames. Then, the feature map features of each frame are projected using a bird's-eye view (BEV) perspective to obtain the three-dimensional spatial features of each frame. It can be understood that each three-dimensional spatial feature can include width, height, and channels.
[0040] Step S102: After stitching the three-dimensional spatial features of other images in consecutive frames except the current frame into specific fusion features, input them into the target detection model to be trained, and obtain the second type of loss function based on the output of the target detection model and the labeled ground truth data of the current frame.
[0041] In this step, in order to minimize the influence of the current frame during model learning, the current frame can be completely excluded from model training. That is, the server concatenates the 3D spatial features of other images in consecutive frames except the current frame into specific fusion features by channel and inputs them into the object detection model (which can be built using known object detection algorithms and model structures). The output of the object detection model is compared with the labeled ground truth data of the current frame, and a second type of loss function is obtained based on the known loss function type.
[0042] Step S103: Form the final loss function based on the second type of loss function, and train the target detection model using the final loss function.
[0043] In this step, a final loss function is formed based on the second type of loss function to train the above object detection model. Optionally, the first way to form the final loss function is to directly determine the second type of loss function as the final loss function. This training method can avoid the defect of the model relying too much on the current frame and helps to improve the model's 3D object recognition ability.
[0044] The following describes a better method for forming the final loss function (the second method for forming the final loss function). First, after extracting the three-dimensional spatial features of consecutive frames from the above video information, multiple extractions can be performed on consecutive frames. Based on each extraction, multiple frames with equally spaced sequence numbers are determined in the consecutive frames.
[0045] To avoid the drawback of traditional techniques that rely too heavily on the current frame due to the full feature stitching of the current frame and all historical frames, this step uses a method of extracting images at equal intervals within consecutive frames and performing feature stitching separately. Specifically, the server performs multiple extractions within consecutive frames, extracting multiple frames with equally spaced sequence numbers in each extraction process. Preferably, the sequence number intervals for different extractions are different and not less than 2. In practical applications, during each extraction process, the server can use the current frame as the first frame determined for that extraction, and then extract images sequentially in descending time order (i.e., in reverse order along the time axis) according to the sequence number intervals for that extraction. As a preferred solution, the sequence number intervals for these multiple extractions cover integers from 2 to a first value, where the first value is half the number of images in the consecutive frames (which can be represented by N). This avoids missing rich temporal motion information in the video.
[0046] For example, if the current frame's sequence number is 1, and the sequence numbers of each historical frame increase by 1 in descending order of time, with the first frame's sequence number being N, then the frames extracted with a sequence number interval of 2 will have sequence numbers 1, 3, 5, 7...; those extracted with a sequence number interval of 3 will have sequence numbers 1, 4, 7, 10...; and those extracted with a sequence number interval of 4 will have sequence numbers 1, 5, 9, 13... The maximum sequence number interval can be N / 2 (or an integer value). For example, when N is 11, the maximum sequence number interval can be 5, and the corresponding extracted frame sequence numbers would be 1, 6, 11.
[0047] Subsequently, the three-dimensional spatial features of each extracted image are concatenated into the temporal fusion features of that extraction and then input into the target detection model to be trained. The first type of loss function for that extraction is obtained based on the output of the target detection model and the labeled ground truth data of the current frame.
[0048] This step processes the results of each extraction independently. Specifically, the server concatenates the 3D spatial features of the extracted image channel by channel to form the temporal fusion feature for that extraction, and then inputs it into the target detection model to be trained. It can be understood that the width and height of the concatenated temporal fusion feature are equal to the width and height of each of the individual 3D spatial features used for concatenation, and the number of channels is the sum of the number of channels of each 3D spatial feature. Afterward, the server compares the output of the target detection model with the labeled ground truth data of the current frame, and obtains the first type of loss function for that extraction based on known loss function types such as cross-entropy.
[0049] Finally, the first type of loss function extracted from each iteration and the second type of loss function above can be added together (including direct addition and addition after setting weights) to obtain the final loss function for training the object detection model. This training method combines interval extraction and feature concatenation after removing the current frame, which can more effectively avoid the model's over-reliance on the current frame and further improve the model's 3D object recognition capability.
[0050] To further improve model performance, the 3D spatial features of each image in consecutive frames can be concatenated by channel into a full fusion feature, which is then input into the object detection model. A third type of loss function is obtained based on the output of the object detection model and the labeled ground truth data of the current frame. The first, second, and third type loss functions extracted multiple times are then added together to obtain the final loss function (a third method of forming the final loss function), which is then used to train the object detection model. This approach further utilizes the full temporal motion information in the video, helping to further improve the object detection capability.
[0051] After the model training is complete, it can be used for 3D object detection. Specifically, the detection server can extract the 3D spatial features of each frame in the video information where the last frame is the frame to be detected. The 3D spatial features of each frame are then concatenated by channel and input into the trained object detection model to obtain the detection results in the frame to be detected. The detection results can include at least one of the following data: target type, target position, target pose, and target size.
[0052] Figure 2 This is a schematic diagram of the model structure and training of an embodiment of the present invention. See [link / reference]. Figure 2 Each consecutive frame in the video information forms a feature map after passing through a feature extraction model. After BEV projection processing, the 3D spatial features of each frame are obtained. The 3D spatial features of each frame are processed in three parts. In the first part, all features are concatenated to obtain a full fused feature. This fused feature is then calculated by the object detection model and compared with the labeled ground truth to form a third-type loss function. In the second part, multiple equally spaced frame number extractions are performed. For each extracted image, the 3D spatial features are concatenated, and the concatenated features are input into the object detection model for calculation. The output is then compared with the labeled ground truth to form a first-type loss function. This process yields one first-type loss function for each extraction. In the third part, features from images other than the current frame are concatenated to obtain specific fused features. This fused feature is then calculated by the object detection model and compared with the labeled ground truth to form a second-type loss function. The first, second, and third-type loss functions are summed to obtain the final loss function, which can be used to train the object detection model.
[0053] According to the technical solution of the present invention, a three-dimensional target detection method based on multiple temporal information fusion methods is proposed. By forcing the model to be given different types of input sequences, the model is forced to learn more comprehensive and richer temporal information, thereby achieving more efficient temporal information fusion.
[0054] In this embodiment of the invention, a target detection method executed by a detection server is further provided. Specifically, the detection server extracts the three-dimensional spatial features of each frame in the video information where the last frame is the frame to be detected. The three-dimensional spatial features of each frame are then concatenated and input into a trained target detection model to obtain the detection result in the frame to be detected. The target detection model is trained according to the aforementioned target detection model training method. Exemplarily, the detection result includes: target type, and also includes at least one of the following data: target position, target pose, and target size.
[0055] The following describes a specific embodiment of the present invention; see [link to specific embodiment]. Figure 2 .
[0056] Pure vision-based 3D (three-dimensional) object detection technology extracts features from input image information through neural network models to obtain the 3D pose information of target obstacles in the scene, thereby realizing perception, control, and planning in autonomous driving scenarios. In recent years, video stream-based 3D object detection methods have developed rapidly and achieved good detection results. Unlike traditional single-frame image input, these methods add historical frame images to the detection information of the current frame, enabling the model to mine the motion characteristics of target objects and capture temporal information. However, existing video stream 3D object detection methods usually simply stitch together and fuse the features of multiple input frames. This crude approach makes the model overly biased towards the information of the current input frame, failing to efficiently mine temporal motion features, thus failing to maximize the advantages of video stream 3D detection methods. To solve this problem, this embodiment proposes a 3D object detection method based on multi-temporal information fusion. By forcing the model to be given different types of input sequences, the model is forced to learn more comprehensive and richer temporal information, achieving more efficient temporal information fusion.
[0057] Existing 3D object detection methods for video streams often simply concatenate the features of each input frame along the channel dimension to achieve multi-frame information fusion. This fusion approach often leads the model to overemphasize certain video frames with more obvious features, such as focusing too much on the current frame while neglecting historical frames. This is because the ground truth labels only correspond to the current frame, making the current frame's input information contain the most and easiest-to-learn geometric features. Therefore, the model becomes overly reliant on the easily learned information of the current frame, ignoring historical frames that also contain rich geometric motion information. This prevents the model from fully leveraging the advantages of multi-frame video stream detection and from achieving optimal multi-frame feature fusion.
[0058] Specifically, the input video information includes images of each frame, I t For the current frame image, I t-1 To I t-(N-1) These are historical frame images, where t, t-1, ... represent frame numbers. Each input image is first processed by a convolutional neural network (i.e., the feature extraction model described above) to extract features, resulting in a feature matrix (i.e., feature map features) for each frame. These feature matrices are then projected into the BEV space to obtain a feature matrix in the BEV space (i.e., 3D spatial features). Features projected into the BEV space can better mine feature information in 3D space.
[0059] After obtaining the BEV feature matrices for each frame, existing methods employ a full fusion and concatenation approach. This method simply concatenates the BEV features of all frames along the channel dimension to obtain a larger feature matrix. The fused feature matrix is then used to calculate the target detection model to be trained. The output and the labeled ground truth are compared using a loss function to derive a third-type loss function. This third-type loss function is then used to guide the target detection model in updating its parameters through gradient backpropagation, thus achieving model training. This simplistic fusion method has the following drawbacks: Among the BEV feature matrices of each frame, the feature matrix of the current frame contains the most and easiest-to-learn key information because the labeled ground truth is itself the manually labeled result of the current frame. Therefore, the model can directly utilize the features of the current frame to achieve basic 3D target detection tasks. However, because learning the features of the current frame is too easy and convenient, the model exhibits laziness, i.e., over-reliance on the current frame features while ignoring the less intuitive and difficult-to-learn information from historical frames. This results in the target detection model only achieving slightly higher detection performance than single-frame detection models. In reality, although historical frame information is more difficult to learn, it contains rich geometric motion information that can significantly improve the model's detection performance. Therefore, it is necessary to explicitly guide the model to move away from the easily learned features of the current frame and force it to mine temporal features in historical frames.
[0060] To achieve the above objectives, the essence is to introduce more difficult learning tasks and force the model to learn from them. To this end, one approach is to introduce a frame-by-frame feature concatenation method, which involves extracting frames at equal intervals from all historical frames, resulting in less input available to the model, a more difficult task, and a larger span between frames. For example, a branch with an index interval of 2 will affect I... t I t-2 I t-4 I t-6 The 3D spatial features are concatenated by channel, with the maximum sequence interval being N / 2 or its integer value, thus obtaining the temporal fusion features extracted each time. These extracted temporal fusion features are then input into the object detection model, and the first-class loss function for each extraction is obtained by comparing it with the labeled ground truth. This method forces the model to minimize the more difficult-to-learn loss function, allowing the model to step out of its comfort zone.
[0061] On the other hand, the features of the current frame, which contain the most information, can be directly discarded, and only the features of each historical frame can be concatenated. This forces the model to predict the 3D object pose of the current frame. This means that the model needs to forcibly mine the motion trajectory features of the object between each frame and predict the detection result of the current frame, which has never been seen before, based on the geometric motion rules of historical frames. This prediction method means the highest learning difficulty. The specific fused features obtained by concatenation are input into the object detection model, and the second type of loss function is obtained by comparing them with the labeled ground truth. By minimizing this loss function, the model can obtain a strong ability to mine temporal features, achieving high-accuracy 3D object detection based on video streams. The introduction of this branch can significantly improve the model's utilization of historical frame information, realize the full release of the advantages of video stream detection models, and achieve good detection results. Finally, the final loss function combining the first, second, and third type of loss functions is used to train the model.
[0062] In the technical solution of this embodiment, by introducing an equal-interval fusion stitching scheme and a predictive fusion stitching scheme, the model is forced to mine more difficult but richer temporal features in historical frames, thereby significantly improving the 3D target detection effect.
[0063] It should be noted that the technical solutions of this invention, including the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.
[0064] For the foregoing method embodiments, they are described as a series of actions for ease of description. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, and some steps may actually be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential for implementing the present invention.
[0065] To facilitate better implementation of the above-described solutions of the embodiments of the present invention, related apparatus for implementing the above-described solutions is also provided below.
[0066] Please see Figure 3 As shown, the training device 300 for the target detection model provided in this embodiment of the invention may include: a feature extraction unit 301, a second function construction unit 302, and a training unit 303.
[0067] The feature extraction unit 301 can be used to: acquire video information as input data for model training, and extract the three-dimensional spatial features of consecutive frames in the video information; wherein the consecutive frames include the current frame as the tail frame; the second function construction unit 302 can be used to: stitch together the three-dimensional spatial features of other images in the consecutive frames except the current frame into a specific fusion feature and input it into the target detection model to be trained, and obtain a second type of loss function based on the output result of the target detection model and the labeled ground truth data of the current frame; the training unit 303 can be used to: form a final loss function based on the second type of loss function, and train the target detection model using the final loss function.
[0068] In one embodiment, the training unit 303 may be further configured to: determine the second type of loss function as the final loss function.
[0069] In this embodiment of the invention, the device 300 may further include: a frame extraction unit, configured to: after extracting the three-dimensional spatial features of consecutive frames in the video information, perform multiple extractions in the consecutive frames, and determine multiple frames of images with equally spaced sequence numbers in the consecutive frames according to each extraction; wherein the sequence number intervals of different extractions are different, and the sequence number interval is not less than 2; a first function construction unit, configured to: concatenate the three-dimensional spatial features of the images determined in each extraction into the temporal fusion features of the extraction and input them into the target detection model, and obtain the first type of loss function of the extraction based on the output result of the target detection model and the labeled ground truth data of the current frame; and the training unit 303 may further be configured to: combine the first type of loss function and the second type of loss function of the multiple extractions into the final loss function.
[0070] As a preferred embodiment, the feature extraction unit 301 can be further used to: obtain the feature map features of each frame in the continuous frames using a preset feature extraction model; and perform bird's-eye view projection processing on the feature map features to obtain the three-dimensional spatial features.
[0071] Preferably, the frame extraction unit can be further configured to: in each extraction, take the current frame as the first frame image determined in this extraction, and extract images sequentially in descending order of time from the current frame according to the sequence number interval of this extraction.
[0072] In specific applications, the interval between the multiple extractions covers an integer from 2 to a first value, where the first value is half the number of images in the consecutive frames.
[0073] In practical applications, the device 300 may further include: a third function construction unit, used to stitch together the three-dimensional spatial features of each image in the consecutive frames into full fusion features and input them into the target detection model, and obtain a third type of loss function based on the output result of the target detection model and the labeled ground truth data of the current frame; and the training unit 303 may further be used to: add the first type of loss function, the second type of loss function and the third type of loss function extracted multiple times to obtain the final loss function.
[0074] Furthermore, in this embodiment of the invention, the labeled truth data includes: target type, and also includes at least one of the following data: target position, target pose, and target size.
[0075] According to the technical solution of the present invention, a three-dimensional target detection method based on multiple temporal information fusion methods is proposed. By forcing the model to be given different types of input sequences, the model is forced to learn more comprehensive and richer temporal information, thereby achieving more efficient temporal information fusion.
[0076] The present invention further provides a target detection device, which can be used to: extract the three-dimensional spatial features of each frame image in video information where the last frame is the frame to be detected; concatenate the three-dimensional spatial features of each frame image and input them into a trained target detection model to obtain the detection result in the frame to be detected. The target detection model is trained using the aforementioned target detection model training device.
[0077] Figure 4 An exemplary system architecture 400 is shown, which can be used to train a target detection model according to embodiments of the present invention or to train a target detection model.
[0078] like Figure 4 As shown, system architecture 400 may include terminal devices 401, 402, and 403, network 404, and server 405 (this architecture is merely an example; the components included in a specific architecture may be adjusted according to the specific application). Network 404 serves as the medium for providing a communication link between terminal devices 401, 402, and 403 and server 405. Network 404 may include various connection types, such as wired or wireless communication links or fiber optic cables.
[0079] Users can use terminal devices 401, 402, and 403 to interact with server 405 via network 404 to receive or send messages, etc. Various communication client applications, such as model training applications (for example only), can be installed on terminal devices 401, 402, and 403.
[0080] Terminal devices 401, 402, and 403 can be various electronic devices with displays that support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0081] Server 405 can be a server that provides various services, such as a backend server that supports model training applications operated by users using terminal devices 401, 402, and 403 (for example only). The backend server can process received model training requests and other related information, and feed back the processing results (such as a trained object detection model – for example only) to terminal devices 401, 402, and 403.
[0082] It should be noted that the training method of the target detection model provided in the embodiments of the present invention is generally executed by the server 405, and correspondingly, the training device of the target detection model is generally set in the server 405.
[0083] It should be understood that Figure 4 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0084] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the training method for the target detection model provided by the present invention.
[0085] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system 500 suitable for implementing an electronic device according to embodiments of the present invention. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.
[0086] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the computer system 500. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0087] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.
[0088] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit 501, it performs the functions defined in the system of this invention.
[0089] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0090] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0091] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including a feature extraction unit, a second function construction unit, and a training unit. The names of these units do not necessarily limit the specific unit; for example, the feature extraction unit can also be described as "a unit that provides three-dimensional spatial features to the second function construction unit."
[0092] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to perform the following steps: acquiring video information as input data for model training; extracting three-dimensional spatial features of consecutive frames in the video information; wherein the consecutive frames include the current frame as the tail frame; concatenating the three-dimensional spatial features of other images in the consecutive frames (excluding the current frame) into specific fusion features and inputting them into a target detection model to be trained; obtaining a second type of loss function based on the output of the target detection model and the labeled ground truth data of the current frame; forming a final loss function based on the second type of loss function; and training the target detection model using the final loss function.
[0093] In the technical solution of this invention embodiment, in order to avoid the shortcomings of existing current frame and historical frame feature fusion that overly favor the current frame and ignore historical frames, this invention adopts the following method to overcome this: the three-dimensional spatial features of historical frames other than the current frame in continuous frames are spliced together, and a second type of loss function is constructed based on the spliced features to form the final loss function to train the target detection model. This method avoids the defect of the model overly relying on the current frame by abandoning the traditional current frame and historical frame feature fusion method, which helps to improve the accuracy of three-dimensional target recognition. More preferably, this embodiment of the invention combines the above method with another feature fusion method, namely, multiple extractions are performed in a continuous frame including the current frame and historical frames. Based on each extraction, multiple frames of images with equally spaced sequence numbers are determined in the continuous frame, wherein the sequence number interval of each extraction is different and not less than 2. Then, the three-dimensional spatial features of the images determined in each extraction are stitched together and a first type of loss function is constructed based on the stitched features. Finally, the first type of loss function and the second type of loss function are combined to generate the final loss function to train the target detection model. This combination scheme, by abandoning the traditional feature fusion method of current frame and historical frame and retaining the rich temporal motion information in the current frame and historical frame, more effectively avoids the defect of the model over-reliance on the current frame, which helps to further improve the three-dimensional target recognition capability of the model.
[0094] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A training method for an object detection model, characterized in that, include: The video information is acquired as input data for model training, and the three-dimensional spatial features of consecutive frames in the video information are extracted; wherein, the consecutive frames include the current frame as the tail frame; Multiple extractions are performed in the continuous frames, and multiple frames of images with equally spaced sequence numbers are determined based on each extraction; wherein the sequence number intervals of different extractions are different, and the sequence number intervals are not less than 2; the three-dimensional spatial features of the images determined in each extraction are concatenated into the temporal fusion features of that extraction and then input into the target detection model to be trained; the first type of loss function for that extraction is obtained based on the output of the target detection model and the labeled ground truth data of the current frame. The three-dimensional spatial features of other images in the consecutive frames, excluding the current frame, are concatenated into specific fusion features and then input into the target detection model to be trained. The second type of loss function is obtained based on the output of the target detection model and the labeled ground truth data of the current frame. The first type of loss function and the second type of loss function extracted multiple times are combined into a final loss function, and the target detection model is trained using the final loss function.
2. The method according to claim 1, characterized in that, The extraction of three-dimensional spatial features from consecutive frames in the video information includes: The feature map features of each frame in the continuous frames are obtained using a preset feature extraction model; The feature map features are subjected to bird's-eye view projection processing to obtain the three-dimensional spatial features.
3. The method according to claim 1, characterized in that, The step of determining multiple frames of images with equally spaced sequence numbers in the consecutive frames based on each extraction includes: In each extraction, the current frame is taken as the first frame image determined in this extraction, and images are extracted sequentially in descending order of time from the current frame according to the extraction sequence number interval.
4. The method according to claim 1, characterized in that, The sequence number intervals of the multiple extractions cover integers from 2 to a first value, where the first value is half the number of images in the consecutive frames.
5. The method according to claim 1, characterized in that, The method further includes: concatenating the three-dimensional spatial features of each image in the consecutive frames into a full fusion feature and inputting it into the target detection model; obtaining a third type of loss function based on the output of the target detection model and the labeled ground truth data of the current frame; and... The step of combining the first type of loss function and the second type of loss function extracted multiple times into a final loss function includes: The first type of loss function, the second type of loss function, and the third type of loss function extracted multiple times are added together to obtain the final loss function.
6. The method according to claim 1, characterized in that, The labeled true data includes: target type, and also includes at least one of the following: target position, target pose, and target size.
7. A target detection method, characterized in that, include: The three-dimensional spatial features of each frame in the video information where the last frame is the frame to be detected are extracted. These features are then concatenated and input into the trained target detection model to obtain the detection result in the frame to be detected. The target detection model is trained according to any one of the methods described in claims 1-6.
8. The method according to claim 7, characterized in that, The detection results include: target type, and also include at least one of the following data: target position, target attitude, and target size.
9. A training device for an object detection model, characterized in that, include: The feature extraction unit is used to acquire video information as input data for model training and extract three-dimensional spatial features of consecutive frames in the video information; wherein, the consecutive frames include the current frame as the tail frame; A frame extraction unit is used to extract multiple frames in the continuous frames, and determine multiple frames of images with equally spaced sequence numbers in the continuous frames according to each extraction; wherein the sequence number interval is different for different extractions, and the sequence number interval is not less than 2; the three-dimensional spatial features of the images determined by each extraction are concatenated into the temporal fusion features of the extraction and then input into the target detection model to be trained; the first type of loss function of the extraction is obtained based on the output result of the target detection model and the labeled ground truth data of the current frame. The second function construction unit is used to stitch together the three-dimensional spatial features of other images in the continuous frames except the current frame into specific fusion features and input them into the target detection model to be trained. Based on the output of the target detection model and the labeled ground truth data of the current frame, a second type of loss function is obtained. The training unit is used to combine the first type of loss function and the second type of loss function extracted multiple times into a final loss function, and to train the target detection model using the final loss function.
10. A target detection device, characterized in that, Used for: extracting the three-dimensional spatial features of each frame image in the video information where the last frame is the frame to be detected, stitching the three-dimensional spatial features of each frame image together and inputting them into the trained target detection model to obtain the detection result in the frame to be detected; wherein, the target detection model is trained by the device according to claim 9.
11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.
