A method, apparatus and device for 3D target detection in continuous RGB images

Abstract

This invention discloses a method, apparatus, device, and storage medium for 3D object detection in continuous RGB images. The method includes: detecting target objects in video frames using a two-dimensional detection model to obtain corresponding two-dimensional detection boxes; cropping a first frame image based on the two-dimensional detection boxes to obtain an object image; inputting the object image into a three-dimensional initialization detection model to obtain a first model matrix and a first scaling factor, and calculating first projection coordinates according to a first formula; inputting the first projection coordinates into a model matrix regression network to obtain a second model matrix and a second scaling factor; cropping a second frame image using the two-dimensional detection boxes and inputting it into the main network to obtain a transformation matrix; calculating second projection coordinates according to a second formula; and calculating a third model matrix according to a third formula. This method avoids semantic ambiguity in the annotation of some three-dimensional objects and reduces detection box jitter.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus and device for 3D target detection in continuous RGB images. Background Technology

[0002] Object detection has always been a fundamental research area in computer vision. In 2D object detection, a wealth of mature technologies already exist. Meanwhile, in 3D space, numerous depth- or point cloud-based object detection techniques are widely used in fields such as autonomous driving. However, 3D object detection often relies on specific hardware or sensors, or multiple cameras, to acquire the object's depth or point cloud data. While these solutions can achieve higher accuracy, they also significantly limit the application scenarios of these algorithms.

[0003] In 3D object detection, the annotation of the detection bounding box in the 3D space of an object has certain semantic ambiguities (for example, it is difficult to define the front of a mineral water bottle), which brings great difficulties to model training. In real-time object detection tasks, it is difficult to ensure the consistency of adjacent frame bounding boxes, resulting in obvious jitter. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose a 3D target detection method, apparatus and device for continuous RGB images, which aims to solve the problem of semantic ambiguity in the annotation of existing three-dimensional objects.

[0005] To achieve the above objectives, the present invention provides a 3D target detection method applied to continuous RGB images, the method comprising:

[0006] The video frame is used to detect target objects by a two-dimensional detection model, and a two-dimensional detection box corresponding to the target object is obtained.

[0007] The object image is obtained by cropping the first frame image containing the target object according to the two-dimensional detection box.

[0008] The object image is input into the three-dimensional initialization detection model to obtain the first model matrix and the first scaling factor under the specified camera matrix and projection matrix, and the first projection coordinates are calculated according to the first formula.

[0009] The first projection coordinates are input into the model matrix regression network to obtain the second model matrix and the second scaling factor under the camera matrix and the projection matrix;

[0010] The second frame image is cropped using the two-dimensional detection box and then input into the main network to obtain the transformation matrix;

[0011] The second projected coordinates are calculated according to the second formula based on the second model matrix, the second scaling factor, and the transformation matrix, and the third model matrix is ​​calculated according to the third formula based on the second model matrix and the transformation matrix.

[0012] Preferably, the network structure of the model matrix regression network includes a single-layer or multi-layer fully connected network.

[0013] Preferably, the first projected coordinates include the coordinates of eight corner points and the coordinates of a geometric center.

[0014] Preferred options also include:

[0015] The first projection coordinates are reprojected using a first loss function; and,

[0016] The second projection coordinates are reprojected using the second loss function.

[0017] Preferably, the first formula is in, Let V represent the first projection coordinates, V represent the camera matrix, P represent the projection matrix, and M represent the projection coordinates. 0 S represents the first model matrix. 0 x,y,z C represents the first scaling factor. x,y,z Represents three-dimensional coordinates.

[0018] Preferably, the second formula is in, Let V represent the second projection coordinates, V represent the camera matrix, P represent the projection matrix, and M represent the projection coordinates. trans Let M represent the transformation matrix. t-1 S represents the second model matrix. t-1 x,y,z C represents the second scaling factor. x,y,z Represents three-dimensional coordinates.

[0019] Preferably, the third formula is M t =M trans M t-1 , of which M t M represents the third model matrix. trans Let M represent the transformation matrix. t-1 This represents the second model matrix.

[0020] To achieve the above objectives, the present invention also provides a 3D target detection device for use in continuous RGB images, the device comprising:

[0021] The target detection unit is used to detect target objects in video frames using a two-dimensional detection model, and obtain a two-dimensional detection box corresponding to the target object.

[0022] The image cropping unit is used to crop the first frame image containing the target object according to the two-dimensional detection box to obtain the object image;

[0023] The first processing unit is used to input the object image into a three-dimensional initialization detection model, obtain a first model matrix and a first scaling factor under a specified camera matrix and projection matrix, and calculate the first projection coordinates according to a first formula.

[0024] The second processing unit is used to input the first projection coordinates into the model matrix regression network to obtain the second model matrix and the second scaling factor under the camera matrix and the projection matrix.

[0025] The third processing unit is used to crop the second frame image through the two-dimensional detection box and input it into the main network to obtain the transformation matrix;

[0026] The fourth processing unit is used to calculate the second projected coordinates based on the second model matrix, the second scaling factor, and the transformation matrix according to the second formula, and to calculate the third model matrix based on the second model matrix and the transformation matrix according to the third formula.

[0027] To achieve the above objectives, the present invention also proposes an apparatus comprising a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a 3D target detection method applied to a continuous RGB image as described in the above embodiments.

[0028] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of a 3D target detection method applied to continuous RGB images as described in the above embodiments.

[0029] Beneficial effects:

[0030] The above scheme obtains the model matrix of the detection box in the current frame under the specified camera matrix and projection matrix by supervising the transformation matrix relative to the camera coordinate system coordinates of the previous frame. This can avoid some semantic ambiguity problems in the annotation of 3D objects and reduce the jitter of the detection box.

[0031] The above scheme avoids the problem that the camera matrix and projection matrix of the labeled data cannot be directly supervised by specifying the camera matrix and projection matrix and directly supervising the coordinates after projection.

[0032] The above scheme, by monitoring the changes in the coordinates of the 3D detection boxes projected onto the two-dimensional space of objects in adjacent frames, fits the transformation matrix of the object's 3D detection box in the camera coordinate system, which can effectively ensure the consistency of consecutive detection frames and improve the stability of the detection box. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a flowchart illustrating a 3D target detection method applied to continuous RGB images according to an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the process of detecting the first frame of a target object to obtain the first projected coordinates, according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram illustrating the process of detecting the target object in each frame after the first frame to obtain the model matrix, according to an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of a 3D target detection device applied to continuous RGB images, provided as an embodiment of the present invention.

[0038] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0041] The present invention will be described in detail below with reference to the embodiments.

[0042] In existing 3D object detection, the following problems still exist: the annotation of 3D object detection boxes is more difficult and costly than that of 2D objects, and there are few publicly available datasets; when annotating data, the camera view matrix and projection matrix are not the same, but in actual use cases, the camera matrix and projection matrix often need to be fixed, making it impossible to directly supervise both during training.

[0043] Based on this, the present invention uses a monocular detection method to learn the three-dimensional transformation matrix between adjacent frames in a fixed camera coordinate system through reprojection supervision, thereby bypassing the semantic ambiguity of the 3D detection box of isolated frames and reducing the jitter of the detection box; and by specifying the camera matrix and projection matrix, the method of directly supervising the coordinates after projection avoids the problem that the camera matrix and projection matrix of the labeled data cannot be directly supervised.

[0044] Reference Figure 1 The diagram shown is a flowchart of a 3D target detection method applied to continuous RGB images according to an embodiment of the present invention.

[0045] In this embodiment, the method includes:

[0046] S11, Target object detection is performed on the video frame using a two-dimensional detection model to obtain a two-dimensional detection box corresponding to the target object;

[0047] S12, crop the first frame image containing the target object according to the two-dimensional detection box to obtain the object image;

[0048] S13, input the object image into the three-dimensional initialization detection model to obtain the first model matrix and the first scaling factor under the specified camera matrix and projection matrix, and calculate the first projection coordinates according to the first formula;

[0049] S14, input the first projection coordinates into the model matrix regression network to obtain the second model matrix and the second scaling factor under the camera matrix and the projection matrix;

[0050] S15, the second frame image is cropped using the two-dimensional detection box and then input into the main network to obtain the transformation matrix;

[0051] S16, the second projected coordinates are calculated according to the second formula based on the second model matrix, the second scaling factor, and the transformation matrix, and the third model matrix is ​​calculated according to the third formula based on the second model matrix and the transformation matrix.

[0052] In practical implementation, it is necessary to first determine a camera view matrix V and a projection matrix P. The selection of these two matrices can be determined based on the actual usage scenario and experience. Furthermore, let's assume a set of 3D coordinates C in the object coordinate system. x,y,z (The coordinate system can be determined based on the object itself. For example, the plane containing the bottom of a shoe can be designated as the xy-axis plane, its normal as the Z-axis, and the direction of the toe as the x-axis.) This is achieved through the camera view matrix V, projection matrix P, transformation matrix M, and scaling factor S. x,y,z Through formula C x',y' =MVPS x,y,z C x,y,z Projecting onto the image coordinate system yields two-dimensional coordinates C. (x',y') .

[0053] In this embodiment, the network model mainly includes a 2D detection network, a 3D initialization network, a model matrix regression network, and a main network. The 2D detection network is primarily used to obtain 2D bounding boxes for objects in the frame image; the network structure of the 2D detection network can adopt open-source object detection schemes, such as YOLO or Faster R-CNN, and is not limited here. The 3D initialization network is used to fit the 3D bounding boxes of the first frame into the camera coordinate system and obtain the 3D coordinates and model matrix of the 3D bounding boxes. Further, the RGB image of the first frame after cropping the 2D bounding boxes obtained by the 2D detection network is used as the input of the 3D initialization network, and its output is the model matrix M under a pre-specified camera matrix V and projection matrix P. 0 and scaling factor S 0 x,y,z The network structure of the Model Matrix Regression Network includes single-layer or multi-layer fully connected networks. The input to the Model Matrix Regression Network is the projected coordinates of the 3D detection box from the previous frame onto the 2D image coordinate system, including nine coordinates: eight corner points and the geometric center. The output is the model matrix M under pre-specified camera matrix V and projection matrix P. t and scaling factor S t-1 x,y,z The main network takes the RGB three-channel image of the current frame as input and outputs the transformation matrix M from the camera coordinates of the detection box in the previous frame to the camera coordinates in the current frame. trans .

[0054] See Figure 2The diagram illustrates the process of detecting the target object in the first frame to obtain the first projected coordinates. The specific implementation includes: detecting the target object in the video frame using a 2D detection model to obtain a 2D detection bounding box; cropping the video frame based on the 2D detection bounding box to obtain a cropped object image; inputting the cropped object image into a 3D initialization detection model, obtaining the model matrix and scaling factor under a specified camera matrix and projection matrix, and then applying the formula... The coordinates projected onto the two-dimensional image coordinate system can be obtained.

[0055] See Figure 3 The diagram illustrates the process of obtaining the model matrix for each frame after the first frame that detects the target object. The specific implementation includes: projecting the previous frame onto the bounding box coordinates of the two-dimensional image coordinate system. Input the model matrix to the regression network, and obtain the model matrix M under the specified camera matrix V and projection matrix P. t-1 and scaling factor S t-1 x,y,z The current frame image is cropped based on the detection box projected onto the 2D image coordinate system from the previous frame. The cropped image is input into the main network to obtain the transformation matrix M from the camera coordinates of the detection box in the previous frame to the camera coordinates in the current frame. trans According to the formula The coordinates of the 3D detection box projected onto the 2D image coordinate system of the current frame are calculated, according to formula M. t =M trans M t-1 Obtain the model matrix M of the current frame detection box under the specified camera matrix V and projection matrix P. t Repeat the above three steps until the object disappears from the image.

[0056] Furthermore, the method also includes reprojection supervision of the coordinates of the previous frame and reprojection supervision of the camera coordinate transformation between adjacent frames. Assume the projected coordinates of the 3D detection box annotated in the previous frame in the 2D image coordinate system are... M is the output of the model matrix regression network. t-1 and S t-1 x,y,z The calculated 2D projection coordinates of the previous frame for network regression are:

[0057]

[0058] The reprojection of the coordinates from the previous frame is supervised using the following loss function:

[0059]

[0060] Furthermore, the reprojection supervision of camera coordinate transformation between adjacent frames is performed using the following loss function:

[0061]

[0062]

[0063] Reference Figure 4 The diagram shown is a structural schematic of a 3D target detection device applied to continuous RGB images according to an embodiment of the present invention.

[0064] In this embodiment, the device 40 includes:

[0065] The target detection unit 41 is used to detect target objects in video frames using a two-dimensional detection model to obtain a two-dimensional detection box corresponding to the target object.

[0066] Image cropping unit 42 is used to crop the first frame image containing the target object according to the two-dimensional detection box to obtain the object image;

[0067] The first processing unit 43 is used to input the object image into the three-dimensional initialization detection model, obtain the first model matrix and the first scaling factor under the specified camera matrix and projection matrix, and calculate the first projection coordinates according to the first formula.

[0068] The second processing unit 44 is used to input the first projection coordinates into the model matrix regression network to obtain the second model matrix and the second scaling factor under the camera matrix and the projection matrix.

[0069] The third processing unit 45 is used to crop the second frame image through the two-dimensional detection box and input it into the main network to obtain the transformation matrix;

[0070] The fourth processing unit 46 is used to calculate the second projected coordinates based on the second model matrix, the second scaling factor, and the transformation matrix according to the second formula, and to calculate the third model matrix based on the second model matrix and the transformation matrix according to the third formula.

[0071] Each unit module of the device 40 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0072] This invention also provides a device comprising the 3D target detection apparatus for continuous RGB images as described above, wherein the 3D target detection apparatus for continuous RGB images can employ... Figure 4 The structure of the embodiment, correspondingly, can be executed Figure 1The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0073] The device includes: a mobile phone, digital camera, or tablet computer with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0074] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as image playback function), etc.; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0075] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0076] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0077] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The method shown is for 3D object detection in continuous RGB images. The computer-readable storage medium may be a read-only memory, a hard disk, or an optical disk, etc.

[0078] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0079] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0080] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A 3D object detection method applied to a continuous RGB image, characterized in that, The method comprises: performing target object detection on the video frame through a two-dimensional detection model to obtain a two-dimensional detection box corresponding to the target object; cropping the first frame image having the target object according to the two-dimensional detection box to obtain an object image; inputting the object image into a three-dimensional initialization detection model to obtain a first model matrix and a first scaling coefficient under a specified camera matrix and a projection matrix, and calculating a first projection coordinate according to a first formula; wherein the first formula is wherein, represents the first projection coordinates, V represents the camera matrix, P represents the projection matrix, represents the first model matrix, represents the first scaling factor, C x,y,z represents three-dimensional coordinates; inputting the first projection coordinate into a model matrix regression network to obtain a second model matrix and a second scaling coefficient under the camera matrix and the projection matrix; cropping the second frame image through the two-dimensional detection box and inputting it into a main network to obtain a transformation matrix; calculating a second projection coordinate according to a second formula based on the second model matrix, the second scaling coefficient and the transformation matrix, and calculating a third model matrix according to a third formula based on the second model matrix and the transformation matrix; wherein the second formula is wherein, denotes the second projection coordinate, V denotes the camera matrix, P denotes the projection matrix, denotes the transformation matrix, denotes the second model matrix, denotes the second scaling factor, C x,y,z denotes a three-dimensional coordinate; wherein the third formula is wherein, denotes the third model matrix, denotes the transformation matrix, denotes the second model matrix.

2. The method for 3D object detection applied in continuous RGB images according to claim 1, characterized in that, the network structure of the model matrix regression network comprises a single-layer or multi-layer fully connected network.

3. The method for 3D object detection applied in continuous RGB images according to claim 1, characterized in that, The first projection coordinate comprises eight corner point coordinates and one geometric center coordinate.

4. The method for 3D object detection applied in continuous RGB images according to claim 1, characterized in that, Further comprising: using a first loss function to perform re-projection supervision on the first projection coordinate; and using a second loss function to perform re-projection supervision on the second projection coordinate. The device comprises:

5. A 3D object detection apparatus applied to a continuous RGB image, characterized in that, a target detection unit configured to perform target object detection on the video frame through a two-dimensional detection model to obtain a two-dimensional detection box corresponding to the target object; an image cropping unit configured to crop the first frame image having the target object according to the two-dimensional detection box to obtain an object image; a first processing unit configured to input the object image into a three-dimensional initialization detection model to obtain a first model matrix and a first scaling coefficient under a specified camera matrix and a projection matrix, and calculate a first projection coordinate according to a first formula; a second processing unit configured to input the first projection coordinate into a model matrix regression network to obtain a second model matrix and a second scaling coefficient under the camera matrix and the projection matrix; wherein the first formula is wherein, denotes the first projection coordinate, V denotes the camera matrix, P denotes the projection matrix, denotes the first model matrix, denotes the first scaling factor, C x,y,z denotes a three-dimensional coordinate; a third processing unit configured to crop the second frame image through the two-dimensional detection box and input it into a main network to obtain a transformation matrix; a fourth processing unit configured to calculate a second projection coordinate according to a second formula based on the second model matrix, the second scaling coefficient and the transformation matrix, and calculate a third model matrix according to a third formula based on the second model matrix and the transformation matrix; a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the 3D target detection method applied to continuous RGB images according to any one of claims 1 to 4. wherein the second formula is wherein, denotes the second projection coordinate, V denotes the camera matrix, P denotes the projection matrix, denotes the transformation matrix, denotes the second model matrix, denotes the second scaling factor, C x,y,z denotes a three-dimensional coordinate; wherein the third formula is wherein, denotes the third model matrix, denotes the transformation matrix, denotes the second model matrix.

6. An apparatus, comprising: A computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the 3D target detection method applied to continuous RGB images according to any one of claims 1 to 4.

7. A computer readable storage medium characterized by ​

Images