Target detection method based on three-dimensional data, device, and medium

Abstract

The present application provides a target detection method based on three-dimensional data, a device, and a medium. The method comprises: constructing a three-dimensional model of a target to be detected; acquiring target images of the three-dimensional model under different rendering parameters; inputting noise and a real image of said target into a generator to generate a first image, and inputting the first image and the target images into a discriminator for training to obtain a trained detection model; and inputting an image under detection into the trained detection model for detection to obtain a target detection result. Virtual cameras are adjusted to perform multi-angle rendering on a target three-dimensional model, thereby ensuring that a target image database has balanced and sufficient data, solving the problem of data imbalance, and improving the detection effect of the detection model.

Description

Target detection method, device and medium based on three-dimensional data Technical Field

[0001] The embodiments of the present application relate to the field of image recognition, and in particular to target detection methods, devices, and media based on three-dimensional data. Background Art

[0002] In the field of computer vision, object detection refers to identifying the types of specific objects in images or videos and determining their locations. However, training object detection models relies on abundant training data. Due to the difficulties in obtaining forensic images, many fields face a shortage of training image data. Insufficient training image data leads to poor model training, which in turn results in poor object detection performance using the trained model.

[0003] Summary of the Invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent. The embodiments of this application provide a target detection method, device and medium based on three-dimensional data, which can ensure balanced and sufficient training data.

[0006] An embodiment of the first aspect of the present application is a method for object detection based on three-dimensional data, comprising:

[0007] Construct a three-dimensional model of the target to be detected;

[0008] Obtain target images of the three-dimensional model of the target to be detected under different rendering parameters;

[0009] Inputting noise and a real image of a target to be detected into a generator of a detection model to generate a first image, inputting the first image and the target image into a discriminator of the detection model for training, and obtaining a trained detection model;

[0010] The image to be detected is input into the trained detection model for detection to obtain the target detection result.

[0011] According to certain embodiments of the first aspect of the present application, obtaining target images of the three-dimensional model of the target to be detected under different rendering parameters includes:

[0012] Set rendering parameters;

[0013] Setting a virtual camera group, wherein a plurality of virtual cameras of the virtual camera group are deployed around the three-dimensional model;

[0014] The target image of the three-dimensional model of the target to be detected under the rendering parameters is obtained through the virtual camera group.

[0015] According to certain embodiments of the first aspect of the present application, obtaining a target image of the three-dimensional model of the target to be detected under rendering parameters by the virtual camera group includes:

[0016] The following steps are performed until the rotation angle of the virtual camera group reaches an angle threshold or the number of target images reaches a number threshold: the virtual camera group is rotated by a preset unit angle, and a target image of the three-dimensional model of the target to be detected under rendering parameters is obtained through the rotated virtual camera group.

[0017] According to some embodiments of the first aspect of the present application, the coordinates of the virtual camera are (x c ,y c , z c );in, z c =rcosθ=0; r is the distance from the virtual camera to the 3D model, θ is the preset parameter, are preset parameters.

[0018] According to certain embodiments of the first aspect of the present application, when the size ratio of the target to be detected to the real image is less than a first ratio threshold, the distance from the virtual camera to the three-dimensional model is increased.

[0019] According to certain embodiments of the first aspect of the present application, obtaining target images of the three-dimensional model of the target to be detected under different rendering parameters includes:

[0020] When the size ratio of the target to be detected to the real image is greater than a second ratio threshold, the three-dimensional model is segmented to obtain segmentation units, and target images of the segmentation units under different rendering parameters are obtained.

[0021] According to certain embodiments of the first aspect of the present application, inputting the first image and the target image into a discriminator of a detection model for training includes:

[0022] When the discriminator determines that the target image is a real image, the pure color background in the target image is removed and the contour line of the target in the target image is marked to obtain a marked image;

[0023] Converting the format of the labeled image to the format of the real image;

[0024] The converted labeled image is superimposed on the real image to obtain a superimposed image.

[0025] According to certain embodiments of the first aspect of the present application, the loss function of the detection model is: L total =LD +L G , L D =-log(D(x))-log(1-D(G(z))),L G = -log(D(G(z))); where L total is the loss function of the detection model, L D is the loss function of the discriminator, L G is the loss function of the generator, D is the discriminator, G is the generator, x is the target image, and z is the noise.

[0026] An embodiment of the second aspect of the present application is an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target detection method based on three-dimensional data as described above is implemented.

[0027] An embodiment of the third aspect of the present application is a computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the target detection method based on three-dimensional data as described above.

[0028] The above scheme has at least the following beneficial effects: by adjusting the virtual camera of the 3D rendering engine, the target 3D model can be rendered without blind spots, and images of the 3D model of the target to be detected at any angle and any distance can be obtained, ensuring that the target image database has balanced and sufficient data, solving the data imbalance problem, which is beneficial to subsequent target detection training and improving the detection effect of the detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0030] FIG1 is a step diagram of a target detection method based on three-dimensional data;

[0031] FIG2 is a sub-step diagram of step S200;

[0032] FIG3 is a diagram showing the positional relationship between the virtual camera group and the three-dimensional model. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0034] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0035] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0036] An embodiment of the present application provides an object detection network, which is applied to the following object detection method based on three-dimensional data.

[0037] 1 , the target detection method based on three-dimensional data includes the following steps:

[0038] Step S100, constructing a three-dimensional model of the target to be detected;

[0039] Step S200, obtaining target images of a three-dimensional model of a target to be detected under different rendering parameters;

[0040] Step S300: Input the noise and the real image of the target to be detected into the generator of the detection model to generate a first image, and input the first image and the target image into the discriminator of the detection model for training to obtain a trained detection model;

[0041] Step S400: Input the image to be detected into the trained detection model for detection to obtain the target detection result.

[0042] In this embodiment, by adjusting the virtual camera of the three-dimensional rendering engine, the target three-dimensional model is rendered without blind spots, and images of the three-dimensional model of the target to be detected at any angle and any distance can be obtained, ensuring that the target image database has balanced and sufficient data, solving the data imbalance problem, and facilitating subsequent target detection training and improving the detection effect of the detection model.

[0043] In step S100 , a three-dimensional model of the target to be detected is constructed.

[0044] On the one hand, the target to be detected can be modeled in three dimensions using three-dimensional modeling software to construct a three-dimensional model of the target to be detected. On the other hand, the three-dimensional model of the target to be detected can be obtained from databases such as PASCAL3D+.

[0045] In step S200 , target images of the three-dimensional model of the target to be detected under different rendering parameters are obtained.

[0046] 2 , obtaining target images of a 3D model of a target to be detected under different rendering parameters includes the following steps:

[0047] Step S210, setting rendering parameters;

[0048] Step S220, setting a virtual camera group;

[0049] Step S230 : obtaining a target image of the three-dimensional model of the target to be detected under rendering parameters through the virtual camera group.

[0050] In step S210 , rendering parameters are set; for example, the rendering background is a solid color background with high contrast to the target model, and the ambient lighting is simulated by a sky box.

[0051] Different rendering parameters can be set to simulate the state of the 3D model in different environments.

[0052] Training samples can be enriched by setting different rendering parameters; in the 3D rendering stage, there are a variety of adjustable parameters, including weather, sunlight intensity, target material, etc. Each time a parameter is adjusted, a large number of training samples can be generated, which has the advantages of low cost and high efficiency.

[0053] In step S220 , a virtual camera group is set.

[0054] As shown in Figure 3, multiple virtual cameras in a virtual camera cluster are deployed around a 3D model. The center block represents the 3D model; the outer blocks represent the cameras. The distance r between the virtual cameras and the 3D model is greater than the maximum of the 3D model's length, width, and height, i.e., r > max(L, W, H).

[0055] In order to solve the data imbalance problem and speed up the rendering speed, multiple cameras are designed to render simultaneously. Assuming that the number of cameras is n (n≤360), the camera group (c1, c2, c3…c n ) can be expressed by the following formula: Among them, θ and is the angle of the virtual camera.

[0056] In step S230 , a target image of the three-dimensional model of the target to be detected under rendering parameters is obtained through the virtual camera group.

[0057] The following steps are performed until the rotation angle of the virtual camera group reaches an angle threshold or the number of target images reaches a number threshold: the virtual camera group is rotated by a preset unit angle, and a target image of the three-dimensional model of the target to be detected under rendering parameters is obtained through the rotated virtual camera group.

[0058] Specifically, the preset unit angle is set to 1 degree. The cameras are evenly distributed on a circle. There must be cameras at coordinates (0, r, 0) and (0, -r, 0). After rendering the first batch of images at the initial coordinates, the cameras at coordinates (0, r, 0) and (0, -r, 0) are deactivated. The remaining cameras rotate around the coordinate axis for rendering. A new batch of target images is obtained with each 1-degree rotation. Rendering stops when the remaining cameras return to their original coordinates.

[0059] For small objects, the distance between the camera and the target model can be increased to obtain small target samples. When the size ratio of the target to be detected to the real image is less than a first ratio threshold, the distance between the virtual camera and the 3D model is increased.

[0060] For close-range targets, the target model can be segmented into multiple segments, and an image sample database can be established for each segmented model. When the size ratio between the target to be detected and the real image is greater than the second ratio threshold, the three-dimensional model is segmented to obtain segmentation units, and the target image of the segmentation unit under different rendering parameters is obtained. For example, if the target to be detected is a humanoid target, a stickman model can be used to establish target image data for multiple trainings, so that the adversarial generative network can generate smaller target images. For incomplete or obscured target images, the human body model can be segmented into three segments from top to bottom, namely the head, upper body, and lower body, and then a target image database can be established for training. In order to achieve close-range target detection, some target-specific feature training samples can also be added, such as pictures of faces, hands, and facial features, for mixed training.

[0061] By performing small target fuzzy detection and large target feature detection, the probability of target detection errors can be effectively reduced.

[0062] For step S300, the noise and the real image of the target to be detected are input into the generator of the detection model to generate a first image, and the first image and the target image are input into the discriminator of the detection model for training to obtain a trained detection model.

[0063] The generator receives a randomly generated Gaussian noise and a real image of the target to be detected, generating a new first image. This first image, along with the target image from the target image database, is then fed into the discriminator for training. The discriminator attempts to classify real data as real and fake data as fake.

[0064] Suppose there are n noise samples {z (1) ,z (2) …z (n)}, n samples from the target image database {x (1) ,x (2) …x (n)}. Calculate the loss function of the discriminator and the generator. The loss function of the detection model is: Ltotal =L D +L G , L D =-log(D(x))-log(1-D(G(z))),L G = -log(D(G(z))); where L total is the loss function of the detection model, L D is the loss function of the discriminator, L G is the loss function of the generator, D is the discriminator, G is the generator, x is the target image, and z is the noise. The parameters of the discriminator and generator are updated sequentially via gradient descent until the generator produces realistic data and the discriminator has difficulty distinguishing between real data and generated data.

[0065] The parameters of the discriminator and generator are continuously updated through an optimization algorithm, so that the images generated by the generator can deceive the discriminator, that is, the discriminator judges the generated images as real. Finally, a trained detection model is obtained.

[0066] When the discriminator identifies the target image as a real image, it uses a thresholding and contour extraction method to remove the solid background from the target image and mark the outline of the target in the target image with a red line, resulting in a marked image. The marked image is then converted to the format of the real image; for example, to a PNG image with an alpha channel. The PNG image is then compressed appropriately to maintain the same resolution and image quality as the real image. The compressed PNG image is then overlaid on the original image to create a composite image with the target outline marked with a red line, resulting in the overlaid image.

[0067] By marking the target with its contour line, the target can be more fully separated from the environment and the spatial volume of the target can be predicted.

[0068] An embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned object detection method based on three-dimensional data when executing the computer program.

[0069] The electronic device may be any intelligent terminal including a computer.

[0070] In general, for the hardware structure of the electronic device, the processor can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0071] The memory can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory and is called by the processor to execute the methods of the embodiments of this application.

[0072] The input / output interface is used to realize information input and output.

[0073] The communication interface is used to realize the communication interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0074] The bus transmits information between the various components of the device (such as the processor, memory, input / output interface, and communication interface). The processor, memory, input / output interface, and communication interface communicate with each other within the device through the bus.

[0075] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for executing the above-mentioned object detection method based on three-dimensional data.

[0076] Those skilled in the art will appreciate that all or some of the steps and systems in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, and the computer-readable medium can include computer storage media (or non-transitory media) and communication media (or temporary media). As known to those skilled in the art, the term computer storage media is included in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data) and is volatile and non-volatile, removable, and non-removable. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium. In the above description of this specification, the reference terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" and the like are intended to mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0077] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0078] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0079] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0081] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. Although the embodiments of the present application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and purpose of the present application, and the scope of the present application is defined by the claims and their equivalents.

[0082] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A target detection method based on three-dimensional data, characterized in that: include: Construct a three-dimensional model of the target to be detected; Obtain target images of the three-dimensional model of the target to be detected under different rendering parameters; Inputting noise and a real image of a target to be detected into a generator of a detection model to generate a first image, inputting the first image and the target image into a discriminator of the detection model for training, and obtaining a trained detection model; The image to be detected is input into the trained detection model for detection to obtain the target detection result.

2. The target detection method based on three-dimensional data according to claim 1, characterized in that: The step of obtaining target images of the three-dimensional model of the target to be detected under different rendering parameters includes: Set rendering parameters; Setting a virtual camera group, wherein a plurality of virtual cameras of the virtual camera group are deployed around the three-dimensional model; The target image of the three-dimensional model of the target to be detected under the rendering parameters is obtained through the virtual camera group.

3. The target detection method based on three-dimensional data according to claim 2, characterized in that: The step of obtaining a target image of the three-dimensional model of the target to be detected under rendering parameters by the virtual camera group includes: The following steps are performed until the rotation angle of the virtual camera group reaches an angle threshold or the number of target images reaches a number threshold: the virtual camera group is rotated by a preset unit angle, and a target image of the three-dimensional model of the target to be detected under rendering parameters is obtained through the rotated virtual camera group.

4. The target detection method based on three-dimensional data according to claim 2, characterized in that: The coordinates of the virtual camera are (x C ,y C , z C );in, Z C = r cosθ = 0; r is the distance from the virtual camera to the 3D model, θ is the preset parameter, are preset parameters.

5. The target detection method based on three-dimensional data according to claim 4, characterized in that: When the size ratio of the target to be detected to the real image is less than a first ratio threshold, the distance between the virtual camera and the three-dimensional model is increased.

6. The target detection method based on three-dimensional data according to claim 1, characterized in that: The step of obtaining target images of the three-dimensional model of the target to be detected under different rendering parameters includes: When the size ratio of the target to be detected to the real image is greater than a second ratio threshold, the three-dimensional model is segmented to obtain segmentation units, and target images of the segmentation units under different rendering parameters are obtained.

7. The target detection method based on three-dimensional data according to claim 1, characterized in that: Inputting the first image and the target image into a discriminator of a detection model for training includes: When the discriminator determines that the target image is a real image, the pure color background in the target image is removed and the contour line of the target in the target image is marked to obtain a marked image; Converting the format of the labeled image to the format of the real image; The converted labeled image is superimposed on the real image to obtain a superimposed image.

8. The target detection method based on three-dimensional data according to claim 1, characterized in that: The loss function of the detection model is: L total =L D +L G , L D =-log(D(x))-log(1-D(G(z))),L G = -log(D(G(z))); where L total is the loss function of the detection model, L D is the loss function of the discriminator, L G is the loss function of the generator, D is the discriminator, G is the generator, x is the target image, and z is the noise.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target detection method based on three-dimensional data as claimed in any one of claims 1 to 8 is implemented.

10. A computer storage medium, characterized in that Computer-executable instructions are stored, and the computer-executable instructions are used to execute the target detection method based on three-dimensional data according to any one of claims 1 to 8.