Sample creation method and device, model training method and device, target detection method and device, equipment and medium

Abstract

The invention provides a sample creation method and device, a model training method and device, a target detection method and device, equipment and a medium, and the sample creation method comprises the steps: obtaining a first sample image collected by a first camera and a second sample image collected by a second camera, cutting an image containing a target object from the second sample image, and obtaining a cut image; performing data enhancement processing on the clipped image by adopting the data enhancement parameter to obtain an enhanced image; covering the preset position of the first sample image with the enhanced image to obtain a third sample image; determining plane position mark information of the target object in the enhanced image according to the data enhancement parameters and the plane position mark information of the target object in the cut image; and converting the plane position marking information of the target object in the enhanced image into spatial position marking information according to the internal reference of the second camera, and converting the spatial position marking information of the target object in the enhanced image into spatial position marking information of the target object in the third sample image according to the internal reference of the first camera.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, equipment and medium for sample creation, model training and target detection. Background Art

[0002] The Position Embedding Transformation for Multi-View 3D Object Detection (PETR) model based on position embedding information perception can detect the spatial position information of the target object, that is, the three-dimensional position information, using the two-dimensional image obtained by shooting the target object with a camera. The target object can be a traffic light, a road sign, a pedestrian, a vehicle, etc. The PETR model can be trained using training data, and the training data is usually a sample image with spatial position marking information. When determining the spatial position marking information of the target object in the sample image, it is necessary to estimate the depth information of the target object. The estimation of the depth information of the target object needs to rely on the original shooting angle and internal parameters of the camera. Therefore, the sample image can only use the original image taken by the camera.

[0003] In the process of implementing the present invention, the inventors discovered that there is a problem of insufficient sample quantity when using original images captured by a camera as sample images for model training. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, device and medium for sample creation, model training and target detection, which can expand the number of samples and solve the problem of insufficient number of samples during model training.

[0005] In a first aspect, an embodiment of the present invention provides a sample creation method, including:

[0006] Acquire a first sample image captured by the first camera and a second sample image captured by the second camera, and crop an image containing a target object from the second sample image to obtain a cropped image;

[0007] Performing data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image;

[0008] Overlaying the enhanced image to a preset position of the first sample image to obtain a third sample image;

[0009] Determine the plane position marking information of the target object in the enhanced image according to the data enhancement parameter and the plane position marking information of the target object in the cropped image;

[0010] The planar position marking information of the target object in the enhanced image is converted into spatial position marking information according to the intrinsic parameters of the second camera, and the spatial position marking information of the target object in the enhanced image is converted into spatial position marking information of the target object in the third sample image according to the intrinsic parameters of the first camera.

[0011] Optionally, acquiring a second sample image captured by a second camera includes:

[0012] Determine the type of object you need;

[0013] A sample image containing a target object of the object type captured by the second camera is acquired to obtain the second sample image.

[0014] According to the category ratio of existing samples, the sample category with a smaller proportion is determined, and the required object type is determined according to the sample category with a smaller proportion, so as to select sample images of target objects containing the required object type, which can solve the problem of unbalanced sample categories.

[0015] Optionally, the first camera is a binocular camera, the first sample image is a dual-view image acquired by the binocular camera, and before the enhanced image is overlaid on a preset position of the first sample image, the method further includes:

[0016] determining a view overlap region of the dual-view image;

[0017] The preset position is determined from an area outside a view overlap area of ​​the dual-view image.

[0018] Overlaying the enhanced image on the non-common visible area of ​​the dual views can avoid destroying the projection consistency.

[0019] Optionally, converting the plane position mark information of the target object in the enhanced image into space position mark information according to the intrinsic parameter of the second camera includes:

[0020] Establishing a first mapping relationship according to the intrinsic parameters of the second camera and the plane position mark information of the target object in the enhanced image, wherein the first mapping relationship is a mapping relationship between the plane position and the spatial position in a second coordinate system, and the second coordinate system is a camera coordinate system of the second camera;

[0021] Solve the first mapping relationship to obtain spatial position marking information of the target object in the enhanced image.

[0022] Optionally, converting the spatial position marking information of the target object in the enhanced image into the spatial position marking information of the target object in the third sample image according to the intrinsic parameter of the first camera includes:

[0023] Establishing a second mapping relationship according to the intrinsic parameters of the first camera and the plane position mark information of the target object in the enhanced image, wherein the second mapping relationship is a mapping relationship between the plane position and the spatial position in a first coordinate system, and the first coordinate system is a camera coordinate system of the first camera;

[0024] The second mapping relationship is solved in combination with the first mapping relationship to obtain the spatial position marking information of the target object in the third sample image.

[0025] Optionally, performing data enhancement processing on the cropped image includes:

[0026] At least one of flipping, scaling and translating is performed on the cropped image.

[0027] In a second aspect, the model training method provided by the embodiment of the present invention includes:

[0028] Acquire training data, wherein the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information;

[0029] Training a preset model based on the training data to obtain a target detection model for detecting a target object;

[0030] The third sample image is created according to the sample creation method described in any embodiment of the present invention.

[0031] In a third aspect, an embodiment of the present invention provides a target detection method, including:

[0032] Acquire the image to be detected;

[0033] Detecting the image to be detected using a target detection model to obtain spatial position detection information of the target object;

[0034] The target detection model is trained according to the model training method described in any embodiment of the present invention.

[0035] In a fourth aspect, an embodiment of the present invention provides a sample creation device, comprising:

[0036] A cropping module, used for acquiring a first sample image captured by the first camera and a second sample image captured by the second camera, and cropping an image containing a target object from the second sample image to obtain a cropped image;

[0037] An enhancement module, used for performing data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image;

[0038] A covering module, used for covering the enhanced image to a preset position of the first sample image to obtain a third sample image;

[0039] a determination module, configured to determine the plane position marking information of the target object in the enhanced image according to the data enhancement parameter and the plane position marking information of the target object in the cropped image;

[0040] a conversion module, configured to convert the planar position marking information of the target object in the enhanced image into spatial position marking information according to the intrinsic parameters of the second camera, and to convert the spatial position marking information of the target object in the enhanced image into spatial position marking information of the target object in the third sample image according to the intrinsic parameters of the first camera.

[0041] In a fifth aspect, the model training device provided by the embodiment of the present invention includes:

[0042] A data acquisition module, used to acquire training data, wherein the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information;

[0043] A training module, used to train a preset model based on the training data to obtain a target detection model for detecting a target object;

[0044] The third sample image is created according to the sample creation method described in any embodiment of the present invention.

[0045] In a sixth aspect, an object detection device provided by an embodiment of the present invention includes:

[0046] An image acquisition module, used for acquiring an image to be detected;

[0047] A detection module, used to detect the image to be detected using a target detection model to obtain spatial position detection information of the target object;

[0048] The target detection model is trained according to the model training method described in any embodiment of the present invention.

[0049] In the seventh aspect, an electronic device provided by an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the sample creation method as described in any embodiment of the present invention is implemented; or, when the processor executes the program, the model training method as described in the embodiment of the present invention is implemented; or, when the processor executes the program, the target detection method as described in the embodiment of the present invention is implemented.

[0050] In an eighth aspect, a computer-readable storage medium is provided in an embodiment of the present invention, on which a computer program is stored, characterized in that when the program is executed by a processor, the sample creation method as described in any embodiment of the present invention is implemented; or, when the program is executed by a processor, the model training method as described in the embodiment of the present invention is implemented; or, when the program is executed by a processor, the target detection method as described in the embodiment of the present invention is implemented.

[0051] The scheme of the embodiment of the present invention creates a new sample image through data enhancement technology. In the process of cropping and constructing an enhanced image from an original image and overlaying the enhanced image on another original image, key information is retained. The key information includes data enhancement parameters, planar position marking information of the target object in the original image and the intrinsic parameters of the camera. After the original image is changed to obtain a new sample image, the position conversion can be accurately performed according to the retained key information, so as to correctly obtain the spatial position of the target object in the new sample image, that is, the target object in the new sample image still has the correct spatial position marking information. Therefore, the new sample image can also be used as a sample for model training, thereby expanding the number of samples and solving the problem of insufficient number of model training samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flow chart of a sample creation method provided by an embodiment of the present invention;

[0054] Figure 2a is another flow chart of the sample creation method provided by an embodiment of the present invention;

[0055] Figure 2b is an example diagram of a sample creation method provided by an embodiment of the present invention;

[0056] Figure 3 It is a flowchart of the model training method provided by an embodiment of the present invention;

[0057] Figure 4 is a flow chart of a target detection method provided by an embodiment of the present invention;

[0058] Figure 5 is a structural schematic diagram of a sample creation device provided by an embodiment of the present invention;

[0059] Figure 6 is a structural schematic diagram of a model training device provided by an embodiment of the present invention;

[0060] Figure 7 is a structural schematic diagram of a target detection device provided by an embodiment of the present invention;

[0061] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0063] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] Figure 1 It is a flow chart of the sample creation method provided in the embodiment of the present invention. The sample creation method provided in the embodiment of the present invention can be applied to scenes such as autonomous driving, robot vision, and security monitoring that require the use of images captured by a camera to detect the spatial position of a target object. The sample creation method can be executed by the sample creation device provided in the embodiment of the present invention, and the sample creation device can be implemented in software and / or hardware. In a specific embodiment, the sample creation device can be integrated into an electronic device, and the electronic device can be a computer, a server, etc. The following embodiment is described by taking the sample creation device integrated into an electronic device as an example, refer to Figure 1 The sample creation method of this embodiment may specifically include the following steps:

[0065] Step 101 : acquiring a first sample image captured by a first camera and a second sample image captured by a second camera, and clipping an image containing a target object from the second sample image to obtain a clipped image.

[0066] The first camera and the second camera may be the same camera or different cameras, and the first camera and the second camera are used to distinguish different sample image acquisition devices. The first sample image is the original image obtained by the first camera shooting the target scene, and the second sample image is the original image obtained by the second camera shooting the target scene. The target scene may be a scene where target detection is required, for example, the target scene may be an autonomous driving scene, a robot vision scene, a security monitoring scene, etc. The target scene shot by the first camera and the second camera may be the same type of scene. The target object may be an object that needs to be located or detected in the target scene, and the target object may be a static object or a dynamic object. For example, the target object may be a traffic light, a pedestrian, a vehicle, a road sign, etc. For example, the first sample image may be the original image obtained by camera A shooting a certain autonomous driving scene, and the second sample image may be the original image obtained by camera B shooting another autonomous driving scene.

[0067] Since it is necessary to crop part of the image from the second sample image and perform data enhancement on the detected image to create a new sample, in order to ensure that the new sample is a valid sample, that is, the new sample includes the target object, in this embodiment, when acquiring the second sample image, it is preferred to use a sample image containing the target object, that is, the second sample image preferably includes the target object, and the second sample image may include one or more target objects. The first sample image may include the target object or may not include the target object, which is not specifically limited here. After obtaining two sample images, the position of the target object in the second sample image can be determined, and according to the position of the target object in the second sample image, the image containing the target object is cropped from the second sample image to obtain a cropped image, that is, the cropped image contains the target object. There can be one or more cropped images.

[0068] Step 102: Perform data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image.

[0069] Data enhancement is a method of changing input data in a certain way to generate more training samples. Data enhancement can reduce the model's excessive reliance on certain features, thereby avoiding overfitting and improving the model's generalization ability and effect. Specifically in this embodiment, data enhancement of an image refers to performing some operations or transformations on the original image to create a new, different image. These operations or transformations can be, for example, rotation, flipping, scaling, translation, cropping, etc. Data enhancement parameters can be parameters of various data enhancement operations, such as the angle of rotation, the direction of flipping, the scale of scaling, the range of cropping, etc. For example, if an image is rotated 30 degrees, then "rotation" is the data enhancement operation, and "30 degrees" is the parameter of this operation. The data enhancement parameters can be pre-set according to actual needs or experience, or they can be randomly set in real time. After the cropped image is subjected to data enhancement processing, the adopted data enhancement parameters can be recorded or retained for subsequent use.

[0070] One or more data augmentation operations may be performed on a cropped image, and the data augmentation operations performed on multiple cropped images may be the same or different.

[0071] Step 103: Overlay the enhanced image to a preset position of the first sample image to obtain a third sample image.

[0072] The preset position may be any position of the first sample image, or a designated position in the first sample image determined according to actual needs or requirements. For example, the preset position may be an edge, a center, a user-designated position, etc. of the first sample image. The third sample image is composed of the first sample image and the enhanced image, that is, the third sample image is an image obtained by replacing the original image at the preset position in the first sample image with the enhanced image. When there are multiple enhanced images, the multiple enhanced images may be used to replace the original images at different positions in the first sample image to generate a rich variety of sample images.

[0073] Step 104 : determining the plane position marking information of the target object in the enhanced image according to the data enhancement parameters and the plane position marking information of the target object in the cropped image.

[0074] The plane position marking information of the target object in the cropped image is used to mark the position of the target object in the cropped image, and can be represented by pixel coordinates. The plane position marking information can be set manually or detected and set by a machine. For example, the plane position marking information of the target object in the cropped image can be the position information of a two-dimensional detection frame, which can also be called a region of interest (ROI) frame, and the position information of the two-dimensional detection frame can be represented by the frame width, height and center point coordinates.

[0075] When data enhancement is performed on an image, the position of the target object in the image will change. After the data enhancement is performed on the image, the image annotation information needs to be updated to reflect the effect of the data enhancement on the position of the target object in the image. Specifically, in this embodiment, the new position mark information of the target object after the enhancement operation can be calculated based on the recorded data enhancement parameters (such as rotation angle, scaling, etc.) and the position of the target object in the cropped image. In this way, no matter how the image is transformed, the position information of the target object will be accurately maintained and tracked.

[0076] For example, the target object is a person, and in the cropped image the person is on the right side of the image. If a horizontal flip data augmentation operation is performed on the image, the person will appear on the left side of the flipped image. This requires redetermining the position of the person in the augmented image based on the executed data augmentation parameters, that is, augmenting the planar position marking information of the target object in the image.

[0077] Step 105: convert the plane position mark information of the target object in the enhanced image into spatial position mark information according to the internal parameters of the second camera, and convert the spatial position mark information of the target object in the enhanced image into spatial position mark information of the target object in the third sample image according to the internal parameters of the first camera.

[0078] The intrinsic parameters of a camera, also known as an intrinsic parameter matrix, are the inherent parameters or properties of the camera. The intrinsic parameters of a camera may include, for example, focal length, principal point, pixel size, distortion parameters, etc. These parameters are determined during the camera calibration process and do not change with changes in the external environment. The intrinsic parameters of different cameras may be different. The intrinsic parameters of a camera can characterize how the camera maps an object from three-dimensional space to a two-dimensional image, and can also characterize how to extract the position information of an object in three-dimensional space from a two-dimensional image. That is, the intrinsic parameters of a camera provide a reference for the conversion between camera coordinates and pixel coordinates.

[0079] Camera coordinates are coordinates in the camera coordinate system. The camera coordinate system (i.e. O-XYZ) is a coordinate system set according to the position and orientation of the camera. It is a three-dimensional coordinate system with the optical center of the camera as the origin and the optical axis as the depth axis. Camera coordinates usually include the following three main parameters:

[0080] X-axis coordinate: represents the distance to the right of the object relative to the camera, that is, the displacement in the left and right directions;

[0081] Y-axis coordinate: represents the distance of the object upward relative to the camera, that is, the displacement in the up and down directions;

[0082] Z-axis coordinate: represents the distance between the object and the camera, that is, the depth information of the object from the camera.

[0083] Pixel coordinates are coordinates in the pixel coordinate system (i.e. o-xy). The origin of the pixel coordinate system is usually located in the upper left corner of the image. The horizontal coordinates represent columns, and the vertical coordinates represent rows. Pixel coordinates refer to the position of each pixel in the image matrix in a two-dimensional image. Generally speaking, pixel coordinates include the following two main parameters:

[0084] x-axis coordinate: represents the distance of this pixel from the left edge of the image;

[0085] Y-axis coordinate: represents the distance of this pixel from the top of the image.

[0086] Specifically in this embodiment, the conversion from plane position to spatial position can be performed first, and then the spatial position conversion under different camera coordinate systems can be performed. When performing the conversion from plane position to spatial position, since the enhanced image comes from the second sample image captured by the second camera, the plane position mark information (i.e., two-dimensional position mark information or pixel coordinates) of the target object in the enhanced image can be first converted into spatial position mark information (i.e., three-dimensional position mark information or camera coordinates) using the internal parameters of the second camera, that is, the spatial position mark information of the target object in the enhanced image can be expressed by the camera coordinates under the camera coordinates of the second camera. Further, when performing the spatial position conversion under different camera coordinate systems, since the enhanced image needs to be overlaid on the first sample image captured by the first camera, the spatial position mark information of the target object in the enhanced image can be converted into the spatial position mark information of the target object in the third sample image using the internal parameters of the first camera, that is, the spatial position mark information of the target object in the third sample image can be expressed by the camera coordinates under the camera coordinates of the first camera.

[0087] Since the third sample image is composed of part of the first sample image and the enhanced image, part of the first sample image is the image of the area in the first sample image that is not covered by the enhanced image, the enhanced image includes the target object, and part of the first sample image may also include the target object. For example, the enhanced image includes a traffic light, and part of the first sample image also includes a traffic light. The spatial position marking information of the target object in the enhanced image can be determined by the method provided above, and the spatial position marking information of the target object in the part of the first sample image can be obtained by converting the plane position marking information of the target object in the part of the first sample image using the internal parameters of the first camera. That is, in the third sample image, each target object can have spatial position marking information.

[0088] The scheme of this embodiment creates a new sample image through data enhancement technology. In the process of cropping and constructing an enhanced image from an original image and overlaying the enhanced image on another original image, key information is retained. The key information includes data enhancement parameters, planar position marking information of the target object in the original image and the intrinsic parameters of the camera. After the original image is changed to obtain a new sample image, the position conversion can be accurately performed according to the retained key information, so as to correctly obtain the spatial position of the target object in the new sample image, that is, the target object in the new sample image still has the correct spatial position marking information. Therefore, the new sample image can also be used as a sample for model training, which expands the number of samples and solves the problem of insufficient number of model training samples.

[0089] Traffic light detection is an important task in autonomous driving, especially in unmanned delivery scenarios. The traffic light perception module in the unmanned delivery vehicle is an important component responsible for detecting and understanding the status of traffic lights. It can help the unmanned delivery vehicle comply with traffic rules on the road and safely share the road with other traffic participants. The following takes the target object as an example to further illustrate the sample creation method provided by the embodiment of the present invention. Figure 2a As shown, the method of this embodiment may include:

[0090] Step 201: Acquire a first sample image captured by a first camera.

[0091] The first camera may be a camera configured in a logistics delivery vehicle. In order to accurately detect traffic lights, a binocular camera is generally used to capture traffic scenes, that is, the first camera may be a binocular camera, and the first sample image may be a dual-view image captured by the binocular camera.

[0092] Step 202, determine the required object type.

[0093] In this embodiment, the sample images will be used to train the target detection model for detecting traffic lights. Traffic lights may include red lights, yellow lights, and green lights. In practical applications, due to traffic environment, selection bias, etc., the sample categories may be unbalanced. For example, in the original images, there are more images including red lights, and fewer images including green lights and yellow lights. Unbalanced sample categories will affect the training effect of the model.

[0094] In order to solve the problem of sample category imbalance, we can determine the sample category with a smaller proportion based on the category ratio of existing samples, and determine the required object type based on the sample category with a smaller proportion. For example, by counting the analogy ratio of existing samples, we find that the sample images including yellow lights account for the smallest proportion, so we can determine that the required traffic light type is yellow.

[0095] Step 203: Acquire a sample image of the target object of the object type captured by the second camera to obtain a second sample image.

[0096] The second camera may also be a binocular camera, and the second sample image may also be a dual-view image captured by the binocular camera. The second camera and the first camera may be the same camera or different cameras. The second sample image includes a target object of a desired type. For example, the second sample image is an image including a yellow light. Of course, the second sample image may also include other types of target objects. For example, the second sample image also includes a green light and / or a red light.

[0097] Step 204: cropping an image containing the target object from the second sample image to obtain a cropped image.

[0098] The position of the target object or the target object of the specified type in the second sample image can be determined, and according to the position of the target object or the target object of the specified type in the second sample image, an image containing the target object or the target object of the specified type is cropped from the second sample image to obtain a cropped image, that is, the cropped image contains the target object or the target object of the specified type, and there can be one or more cropped images. For example, an image including a yellow light, a red light, and / or a green light can be cropped from the second sample image, and the cropped image including the traffic light is the cropped image.

[0099] Step 205: Perform data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image.

[0100] Performing data enhancement on an image refers to performing some operations or transformations on the original image to create a new, different image. In this embodiment, the data enhancement process may include at least one of flipping, scaling, and translation. The data enhancement parameters are parameters of various data enhancement operations, such as flipping direction, scaling factor, translation distance, etc.

[0101] Step 206: determine the view overlap area of ​​the first sample image.

[0102] In this embodiment, since the first sample image is a dual-view image captured by a binocular camera, there is a certain angle between the binocular cameras, and the optical center positions of the binocular cameras are basically coincident, there is a common visible part in the views of the two cameras, that is, there is a view overlap area in the first sample image, and the view overlap area of ​​the first sample image can be determined. Assuming that the first sample image is divided into a left view and a right view, generally speaking, the view overlap area of ​​the first sample image can be the right half image of the left view and the left half image of the right view. When cropping and overlaying images, the common visible part can be avoided as much as possible to avoid projection inconsistency.

[0103] Step 207 : determining a preset position from an area outside the view overlap area of ​​the first sample image.

[0104] In order to avoid destroying the common visible part of the view, the preset position can be determined from the area outside the view overlap area of ​​the first sample image. For example, the position corresponding to the left half image of the left view and / or the right half image of the right view of the first sample image can be determined as the preset position.

[0105] Correspondingly, the second sample image may also include a left view and a right view. The left view of the second sample image may be cut into a left half image and a right half image, and the right view of the second sample image may also be cut into a left half image and a right half image. A cropped image is selected from the four half images obtained by cutting, and data enhancement processing is performed on the cropped image. The cropped image may be selected from the four half images obtained by cutting, and may be selected randomly or according to preset rules. The preset rules may include, for example, selecting an image including a target object, or selecting an image including a specified type of target object. For example, an image including a yellow light may be selected, or an image including a red light and / or a green light may be selected.

[0106] Step 208: Overlay the enhanced image to a preset position of the first sample image to obtain a third sample image.

[0107] The third sample image is composed of the first sample image and the enhanced image, that is, the third sample image is an image obtained by replacing the original image at a preset position in the first sample image with the enhanced image. When there are multiple enhanced images, the multiple enhanced images can be used to replace the original images at different positions in the first sample image to generate rich and diverse sample images.

[0108] The first sample image is represented by P, P = P1, P2, P3, P4. Among them, P1 is the left half image of the left view of the first sample image, P2 is the right half image of the left view of the first sample image, P3 is the left half image of the right view of the first sample image, and P4 is the right half image of the right view of the first sample image. The second sample image is represented by P', P' = P1', P2', P3', P4'. Among them, P1' is the left half image of the left view of the second sample image, P2' is the right half image of the left view of the second sample image, P3' is the left half image of the right view of the second sample image, and P4' is the right half image of the right view of the second sample image. The image selected from P1', P2', P3', P4' is called P i ′, i=1,2,3,4, that is, P i ′ is the cropped image, then P i ' and P can be used to create a third sample image. For example, the selected images are P1' and P2', P1' can be overlaid on P1 after data enhancement processing, and P2' can be overlaid on P4 after data enhancement processing to create a third sample image.

[0109] Step 209: establishing a first mapping relationship according to the intrinsic parameters of the second camera and the plane position mark information of the target object in the enhanced image.

[0110] The first mapping relationship is a mapping relationship between a plane position and a space position in a second coordinate system, and the second coordinate system is a camera coordinate system of the second camera.

[0111] Specifically, the plane position mark information of the target object in the enhanced image can be determined according to the data enhancement parameters and the plane position mark information of the target object in the cropped image, and then a first mapping relationship is established according to the intrinsic parameters of the second camera and the plane position mark information of the target object in the enhanced image. The data enhancement parameters are retained when the cropped image is subjected to data enhancement processing.

[0112] Step 210: Solve the first mapping relationship to obtain spatial position marking information of the target object in the enhanced image.

[0113] That is, the planar position marking information (i.e., two-dimensional position marking information or pixel coordinates) of the target object in the enhanced image is converted into the spatial position marking information (i.e., three-dimensional position marking information or camera coordinates) of the target object in the enhanced image using the first mapping relationship. The spatial position marking information of the target object in the enhanced image can be represented by the camera coordinates under the camera coordinates of the second camera.

[0114] Step 211: establish a second mapping relationship according to the intrinsic parameters of the first camera and the plane position mark information of the target object in the enhanced image.

[0115] The second mapping relationship is a mapping relationship between a plane position and a space position in a first coordinate system, and the first coordinate system is a camera coordinate system of the first camera.

[0116] Step 212: Solve the second mapping relationship in combination with the first mapping relationship to obtain spatial position marking information of the target object in the third sample image.

[0117] The spatial position marking information of the target object in the third sample image may be represented by the camera coordinates under the camera coordinates of the first camera.

[0118] Combine the following Figure 2b The sample creation method provided by the embodiment of the present invention is described by way of example. Figure 2bIt is an example diagram of the sample creation method provided by an embodiment of the present invention. In this embodiment, the target object is a traffic light, and the data enhancement processing performed is flipping, scaling and translation processing. The original sample includes a first sample image P and a second sample image P′, and the target sample includes a third sample image. The planar position marking information of the target object is represented by a two-dimensional box, and the spatial position information of the target object is represented by a three-dimensional box. The entire sample creation process may include three steps: image clipping and blending, two-dimensional box transformation and center point transformation. Among them, the center point transformation includes flip transformation, camera transformation, scaling transformation and translation transformation. The three-dimensional box can be obtained by combining the two-dimensional box transformation and the center point transformation. The following is a detailed introduction to each step:

[0119] (1) Image cropping and blending.

[0120] The first sample image P = P1, P2, P3, P4, the images P1 and P4 of the non-common visible parts taken by the first camera can be used as the positions to be covered, and two half images P are selected from the second sample image P' = P1', P2', P3', P4' i ′, select a P i ′ is enhanced and covered on P1, and another P i ’ is enhanced and overlaid on P4 to obtain a third sample image that does not destroy the projection consistency.

[0121] (2) Two-dimensional box transformation.

[0122] P i The data augmentation operations of ′ include random flipping, random scaling, and random translation. Assume that P i The two-dimensional box of a traffic light in ′ is represented as in, are the horizontal and vertical coordinates of the center point of the original two-dimensional box, is the width and height of the original two-dimensional box.

[0123] First, the two-dimensional frame is randomly flipped, and the two-dimensional frame obtained after flipping is expressed as Where W is the width of the single view image, Indicates the horizontal and vertical coordinates of the center point of the two-dimensional box after flipping. Indicates the width and height of the flipped 2D box.

[0124] Next, the flipped two-dimensional frame is randomly scaled, and the scaled two-dimensional frame is expressed as Where s is the scaling factor, Indicates the horizontal and vertical coordinates of the center point of the two-dimensional box after scaling. Indicates the width and height of the two-dimensional box after scaling.

[0125] Finally, the two-dimensional frame obtained after scaling is randomly translated, and the two-dimensional frame obtained after translation is expressed as If the translation coordinates are And after translation, the image covers P1, then the image coordinate change value is If the translation coordinates are And after translation, the image is overlaid on P4, then the image coordinate change value is but in, Represents the horizontal and vertical coordinates of the center point of the two-dimensional box after translation. Indicates the width and height of the two-dimensional box after translation.

[0126] (3) Center point transformation.

[0127] P i ′ comes from the second camera, assuming P i The three-dimensional coordinates of the center point of a traffic light in the camera coordinate system of the second camera are The original two-dimensional box corresponding to the traffic light is The intrinsic parameter matrix of the second camera is:

[0128]

[0129] Where K represents the intrinsic parameter matrix of the second camera, (f x 、f y ) represents the focal length of the second camera, (c x , c y ) represents the principal point coordinates of the second camera.

[0130] Then the position mapping relationship of the center point of the traffic light in the camera coordinate system of the second camera can be expressed as:

[0131]

[0132] Solving formula 2, we can get the three-dimensional coordinates of the center point of the traffic light in the camera coordinate system of the second camera: That is, the three-dimensional coordinates of the center point of the two-dimensional frame of the traffic light in the camera coordinate system of the second camera are

[0133] For flip transformation:

[0134] From the previous content, we can know that the two-dimensional frame obtained after flipping Assume that the 3D coordinates of the center point of the flipped 2D box are The flip transformation only changes the horizontal axis coordinate value X1, and does not affect the other two axis coordinate values ​​Y1 and Z1. The position mapping relationship after flipping can be expressed as:

[0135]

[0136] Solving formula 3, we can get the three-dimensional coordinates of the center point of the flipped two-dimensional frame: Right now:

[0137]

[0138] For camera transformation:

[0139] Since the enhanced image comes from the second camera, the enhanced image will eventually be overlaid on the first sample image captured by the first camera, so it is necessary to perform camera transformation on the position data.

[0140] Assume that the intrinsic parameter matrix of the first camera is:

[0141]

[0142] Where K′ represents the intrinsic parameter matrix of the first camera, (f x ′、f y ′) represents the focal length of the first camera, (c′ x , c′ y ) represents the coordinates of the principal point of the first camera.

[0143] The two-dimensional box obtained after flipping Assume that the three-dimensional coordinates of the center point of the two-dimensional box after camera transformation are Then the position mapping relationship of the center point of the traffic light in the camera coordinate system of the first camera can be expressed as:

[0144]

[0145] Combining Formula 3 with Formula 6, we can get the three-dimensional coordinates of the center point of the two-dimensional frame after the camera transformation:

[0146]

[0147] For scaling transformation:

[0148] The two-dimensional box obtained after scaling Assume that the three-dimensional coordinates of the center point of the two-dimensional box after scaling transformation are Then the position mapping relationship after scaling can be expressed as:

[0149]

[0150] Combining Formula 6 with Formula 8, we can obtain:

[0151]

[0152] For translation transformation:

[0153] The two-dimensional box obtained after translation is [x3, y3, w3, h3] T =[x2+d x ,y2+d y ,w2,h2] T , assuming that the three-dimensional coordinates of the center point of the two-dimensional box after translation transformation are Then the position mapping relationship after translation can be expressed as:

[0154]

[0155] Combining formula 8 with formula 10, we can obtain:

[0156]

[0157] After obtaining the three-dimensional coordinates of the center point of the three-dimensional frame of the target object in the third sample image, the three-dimensional coordinates of other points in the three-dimensional frame can be determined by combining the two-dimensional frame of the target object in the enhanced image and using a method similar to that of determining the three-dimensional coordinates of the center point of the three-dimensional frame. After determining the three-dimensional coordinates of all points in the three-dimensional frame, the spatial position marking information of the target object in the third sample image can be determined according to the three-dimensional coordinates of all points in the three-dimensional frame.

[0158] The spatial position of an object can be composed of seven degrees of freedom. If the detection frame is axisymmetric, the spatial position can be degenerated into six degrees of freedom, namely the three-dimensional coordinates, length, width and height of the center point of the detection frame. The three-dimensional coordinates, length, width and height of the center point of the detection frame can be used as the spatial position marking information of the target object in the third sample image. After obtaining the third sample image and the spatial position marking information of the target object in the third sample image, the third sample image and the spatial position marking information of the target object in the third sample image can be used as a training sample. That is, in the training sample, the sample data is the third sample image, and the sample label is the spatial position marking information of the target object in the third sample image.

[0159] It should be noted that the above only uses flipping, scaling and translation processing for data enhancement, and the processing order is flipping first, then scaling, and finally translation. In actual applications, more or fewer data enhancement methods can also be used, or the matching order between various data enhancement methods can also be adaptively adjusted according to actual needs.

[0160] The scheme of this embodiment creates a new sample image through data enhancement technology. In the process of cropping and constructing an enhanced image from an original image and overlaying the enhanced image on another original image, key information is retained. The key information includes data enhancement parameters, planar position marking information of the target object in the original image and the intrinsic parameters of the camera. After the original image is changed to obtain a new sample image, the position conversion can be accurately performed according to the retained key information, so as to correctly obtain the spatial position of the target object in the new sample image, that is, the target object in the new sample image still has the correct spatial position marking information. Therefore, the new sample image can also be used as a sample for model training, which expands the number of samples and solves the problem of insufficient number of model training samples.

[0161] Furthermore, a second sample image is selected according to the object type requirement, and data enhancement processing is performed on the image cropped from the second sample image to create a sample, which can balance the sample categories, increase the diversity of training data, and avoid overfitting of model training.

[0162] In addition, when overlaying the enhanced image on the original image, the enhanced image is overlaid on the non-commonly visible area of ​​the original image to avoid destroying the projection consistency and ensure the accuracy of the position marking information determined subsequently.

[0163] The following describes the model training method provided by the embodiment of the present invention. Figure 3 As shown, the model training method of this embodiment may include:

[0164] Step 301: Acquire training data, where the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information.

[0165] The first sample image, the second sample image and the third sample image all have spatial position marking information. That is, the training data includes the original image captured by the camera and the image created using the original image. In the training data, the sample data is the sample image, and the sample label is the spatial position marking information of the target object in the corresponding sample image. That is, the present invention uses labeled samples for model training, which can speed up the model training speed.

[0166] The third sample image is created according to the sample creation method provided in the embodiment of the present invention. For the specific creation process, please refer to the previous embodiment and will not be described again here.

[0167] Step 302: training a preset model based on the training data to obtain a target detection model for detecting the target object.

[0168] The target object can be any object that needs to be detected, including but not limited to traffic lights, pedestrians, vehicles, signs, etc., depending on the label information in the training data.

[0169] In this embodiment, original samples and created samples are used for model training. The number of samples is sufficient, the sample types are balanced, and the training data is rich, which can avoid model overfitting. The trained model can better adapt to various situations, has better generalization ability, and can perform robustly under various conditions.

[0170] The target detection method provided by the embodiment of the present invention is described below. Figure 4 As shown, the target detection method of this embodiment may include:

[0171] Step 401: Acquire an image to be detected.

[0172] The image to be detected may be an image captured by a camera and requiring target detection, and the image to be detected may include one or more images.

[0173] Step 402: Detect the image to be detected using the target detection model to obtain spatial position detection information of the target object.

[0174] The target detection model is trained using the samples and model training method provided by the embodiment of the present invention. The specific process can be referred to the description of the previous embodiment and will not be repeated here. In this embodiment, the spatial position detection information of the target object can be represented by three-dimensional frame information, which can reflect the three-dimensional coordinates, length, width and height of the center point of the target object.

[0175] In this embodiment, a target detection model is used to detect the target object. The target detection model is trained using original samples and created samples. The number of samples is sufficient, the sample types are balanced, the training data is rich, and the model has good generalization ability and is robust. Therefore, during the application process, the model can learn more complex features and patterns, thereby improving the detection ability and effect of the model.

[0176] Figure 5 is a schematic diagram of a structure of a sample creation device provided in an embodiment of the present invention. The sample creation device is suitable for executing the sample creation method provided in an embodiment of the present invention. Figure 5 As shown, the device may specifically include:

[0177] A cropping module 501 is used to obtain a first sample image captured by a first camera and a second sample image captured by a second camera, and crop an image containing a target object from the second sample image to obtain a cropped image;

[0178] An enhancement module 502 is used to perform data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image;

[0179] A covering module 503 is used to cover the enhanced image to a preset position of the first sample image to obtain a third sample image;

[0180] A determination module 504, configured to determine the plane position marking information of the target object in the enhanced image according to the data enhancement parameter and the plane position marking information of the target object in the cropped image;

[0181] The conversion module 505 is used to convert the planar position marking information of the target object in the enhanced image into spatial position marking information according to the internal parameters of the second camera, and to convert the spatial position marking information of the target object in the enhanced image into the spatial position marking information of the target object in the third sample image according to the internal parameters of the first camera.

[0182] In one embodiment, the cropping module 501 obtains the second sample image captured by the second camera, including:

[0183] Determine the type of object you need;

[0184] A sample image containing a target object of the object type captured by the second camera is acquired to obtain the second sample image.

[0185] In one embodiment, the first camera is a binocular camera, the first sample image is a dual-view image acquired by the binocular camera, and the overlay module 503 is further configured to:

[0186] determining a view overlap region of the dual-view image;

[0187] The preset position is determined from an area outside a view overlap area of ​​the dual-view image.

[0188] In one embodiment, the conversion module 505 converts the plane position mark information of the target object in the enhanced image into the space position mark information according to the intrinsic parameter of the second camera, including:

[0189] Establishing a first mapping relationship according to the intrinsic parameters of the second camera and the plane position mark information of the target object in the enhanced image, wherein the first mapping relationship is a mapping relationship between the plane position and the spatial position in a second coordinate system, and the second coordinate system is a camera coordinate system of the second camera;

[0190] Solve the first mapping relationship to obtain spatial position marking information of the target object in the enhanced image.

[0191] In one embodiment, the conversion module 505 converts the spatial position marking information of the target object in the enhanced image into the spatial position marking information of the target object in the third sample image according to the intrinsic parameters of the first camera, including:

[0192] Establishing a second mapping relationship according to the intrinsic parameters of the first camera and the plane position mark information of the target object in the enhanced image, wherein the second mapping relationship is a mapping relationship between the plane position and the spatial position in a first coordinate system, and the first coordinate system is a camera coordinate system of the first camera;

[0193] The second mapping relationship is solved in combination with the first mapping relationship to obtain the spatial position marking information of the target object in the third sample image.

[0194] In one embodiment, the enhancement module 502 performs data enhancement processing on the cropped image, including:

[0195] At least one of flipping, scaling and translating is performed on the cropped image.

[0196] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0197] The device of the embodiment of the present invention creates a new sample image through data enhancement technology. In the process of cropping and constructing an enhanced image from an original image and overlaying the enhanced image on another original image, key information is retained. The key information includes data enhancement parameters, planar position marking information of the target object in the original image and the intrinsic parameters of the camera. After the original image is changed to obtain a new sample image, the position conversion can be accurately performed according to the retained key information, so as to correctly obtain the spatial position of the target object in the new sample image, that is, the target object in the new sample image still has the correct spatial position marking information. Therefore, the new sample image can also be used as a sample for model training, thereby expanding the number of samples and solving the problem of insufficient number of model training samples.

[0198] Figure 6 is a schematic diagram of the structure of a model training device provided by an embodiment of the present invention, such as Figure 6 As shown, the device may include:

[0199] A data acquisition module 601 is used to acquire training data, wherein the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information;

[0200] A training module 602 is used to train a preset model based on the training data to obtain a target detection model for detecting a target object;

[0201] The third sample image is created according to the sample creation method described in any embodiment of the present invention.

[0202] The device of this embodiment uses original samples and created samples for model training. The sample quantity is sufficient, the sample types are balanced, and the training data is rich, which can avoid model overfitting. The trained model can better adapt to various situations, has better generalization ability, and can perform robustly under various conditions.

[0203] Figure 7 is a schematic diagram of a structure of a target detection device provided by an embodiment of the present invention, such as Figure 7 As shown, the device may include:

[0204] The image acquisition module 701 is used to acquire the image to be detected;

[0205] A detection module 702 is used to detect the image to be detected using a target detection model to obtain spatial position detection information of the target object;

[0206] The target detection model is trained according to the model training method described in the embodiment of the present invention.

[0207] The device of this embodiment uses a target detection model to detect the target object. The target detection model is trained using original samples and created samples. The number of samples is sufficient, the sample types are balanced, the training data is rich, and the model has good generalization ability and is robust. Therefore, during the application process, the model can learn more complex features and patterns, thereby improving the detection ability and effect of the model.

[0208] An embodiment of the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the sample creation method provided in any one of the above embodiments is implemented; or, when the processor executes the program, the model training method provided in any one of the above embodiments is implemented; or, when the processor executes the program, the target detection method provided in any one of the above embodiments is implemented.

[0209] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of an electronic device suitable for implementing an embodiment of the present invention. Figure 8The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0210] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the computer system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0211] Typically, the following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0212] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0213] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0214] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0215] The modules and / or units involved in the embodiments of the present invention may be implemented in software or hardware. The modules and / or units described may also be set in a processor, for example, may be described as: a processor including a cropping module, an enhancement module, a covering module, a determination module, and a conversion module; or may be described as: a data acquisition module and a training module; or may be described as: an image acquisition module and a detection module. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0216] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0217] Acquire a first sample image captured by the first camera and a second sample image captured by the second camera, and crop an image containing a target object from the second sample image to obtain a cropped image;

[0218] Performing data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image;

[0219] Overlaying the enhanced image to a preset position of the first sample image to obtain a third sample image;

[0220] Determine the plane position marking information of the target object in the enhanced image according to the data enhancement parameter and the plane position marking information of the target object in the cropped image;

[0221] The planar position marking information of the target object in the enhanced image is converted into spatial position marking information according to the intrinsic parameters of the second camera, and the spatial position marking information of the target object in the enhanced image is converted into spatial position marking information of the target object in the third sample image according to the intrinsic parameters of the first camera.

[0222] Alternatively, when the above one or more programs are executed by a device, the device includes:

[0223] Acquire training data, wherein the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information;

[0224] Training a preset model based on the training data to obtain a target detection model for detecting a target object;

[0225] The third sample image is created according to the sample creation method described in any embodiment of the present invention.

[0226] Alternatively, when the above one or more programs are executed by a device, the device includes:

[0227] Acquire the image to be detected;

[0228] Detecting the image to be detected using a target detection model to obtain spatial position detection information of the target object;

[0229] The target detection model is trained according to the model training method described in the embodiment of the present invention.

[0230] The technical solution of the embodiment of the present invention creates a new sample image through data enhancement technology. In the process of cropping and constructing an enhanced image from an original image and overlaying the enhanced image on another original image, key information is retained. The key information includes data enhancement parameters, planar position marking information of the target object in the original image and the intrinsic parameters of the camera. After the original image is changed to obtain a new sample image, the position conversion can be accurately performed according to the retained key information, so as to correctly obtain the spatial position of the target object in the new sample image, that is, the target object in the new sample image still has the correct spatial position marking information. Therefore, the new sample image can also be used as a sample for model training, thereby expanding the number of samples and solving the problem of insufficient number of model training samples.

[0231] Furthermore, the original samples and created samples are used for model training. The sample quantity is sufficient, the sample types are balanced, and the training data is rich, which can avoid model overfitting. The trained model can better adapt to various situations, have better generalization ability, and can perform robustly under various conditions.

[0232] In addition, the target detection model is used to detect the target object. The target detection model is trained using the original samples and the created samples. The sample quantity is sufficient, the sample types are balanced, the training data is rich, and the model generalization ability is good and robust. Therefore, in the process of application, the model can learn more complex features and patterns, thereby improving the detection ability and effect of the model.

[0233] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a sample creation method as provided in any embodiment of the present application; or, when executed by a processor, implements a model training method as provided in any embodiment of the present application; or, when executed by a processor, implements a target detection method as provided in any embodiment of the present application.

[0234] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0235] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0236] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0237] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A sample creation method, characterized in that: include: Acquire a first sample image captured by the first camera and a second sample image captured by the second camera, and crop an image containing a target object from the second sample image to obtain a cropped image; Performing data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image; Overlaying the enhanced image to a preset position of the first sample image to obtain a third sample image; Determine the plane position marking information of the target object in the enhanced image according to the data enhancement parameter and the plane position marking information of the target object in the cropped image; The planar position marking information of the target object in the enhanced image is converted into spatial position marking information according to the intrinsic parameters of the second camera, and the spatial position marking information of the target object in the enhanced image is converted into spatial position marking information of the target object in the third sample image according to the intrinsic parameters of the first camera.

2. The sample creation method according to claim 1, characterized in that: Acquiring a second sample image acquired by a second camera, including: Determine the type of object you need; A sample image containing a target object of the object type captured by the second camera is acquired to obtain the second sample image.

3. The sample creation method according to claim 1, characterized in that: The first camera is a binocular camera, the first sample image is a dual-view image acquired by the binocular camera, and before the enhanced image is overlaid on a preset position of the first sample image, the method further includes: determining a view overlap region of the dual-view image; The preset position is determined from an area outside a view overlap area of ​​the dual-view image.

4. The sample creation method according to claim 1, characterized in that: The converting the plane position mark information of the target object in the enhanced image into the space position mark information according to the internal parameter of the second camera includes: Establishing a first mapping relationship according to the intrinsic parameters of the second camera and the plane position mark information of the target object in the enhanced image, wherein the first mapping relationship is a mapping relationship between the plane position and the spatial position in a second coordinate system, and the second coordinate system is a camera coordinate system of the second camera; Solve the first mapping relationship to obtain spatial position marking information of the target object in the enhanced image.

5. The sample creation method according to claim 4, characterized in that: The converting the spatial position marking information of the target object in the enhanced image into the spatial position marking information of the target object in the third sample image according to the internal parameter of the first camera includes: Establishing a second mapping relationship according to the intrinsic parameters of the first camera and the plane position mark information of the target object in the enhanced image, wherein the second mapping relationship is a mapping relationship between the plane position and the spatial position in a first coordinate system, and the first coordinate system is a camera coordinate system of the first camera; The second mapping relationship is solved in combination with the first mapping relationship to obtain the spatial position marking information of the target object in the third sample image.

6. The sample creation method according to any one of claims 1 to 5, characterized in that: The performing data enhancement processing on the cropped image comprises: At least one of flipping, scaling and translating is performed on the cropped image.

7. A model training method, characterized in that: include: Acquire training data, wherein the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information; Training a preset model based on the training data to obtain a target detection model for detecting a target object; The third sample image is created according to the sample creation method according to any one of claims 1 to 6.

8. A target detection method, characterized in that: include: Acquire the image to be detected; Detecting the image to be detected using a target detection model to obtain spatial position detection information of the target object; Wherein, the target detection model is trained according to the model training method according to claim 7.

9. A sample creation device, characterized in that: include: A cropping module, used for acquiring a first sample image captured by the first camera and a second sample image captured by the second camera, and cropping an image containing a target object from the second sample image to obtain a cropped image; An enhancement module, used for performing data enhancement processing on the cropped image using data enhancement parameters to obtain an enhanced image; A covering module, used for covering the enhanced image to a preset position of the first sample image to obtain a third sample image; a determination module, configured to determine the plane position marking information of the target object in the enhanced image according to the data enhancement parameter and the plane position marking information of the target object in the cropped image; a conversion module, configured to convert the planar position marking information of the target object in the enhanced image into spatial position marking information according to the intrinsic parameters of the second camera, and to convert the spatial position marking information of the target object in the enhanced image into spatial position marking information of the target object in the third sample image according to the intrinsic parameters of the first camera.

10. A model training device, characterized in that: include: A data acquisition module, used to acquire training data, wherein the training data includes a first sample image, a second sample image, and a third sample image having spatial position marking information; A training module, used to train a preset model based on the training data to obtain a target detection model for detecting a target object; The third sample image is created according to the sample creation method according to any one of claims 1 to 6.

11. A target detection device, characterized in that: include: An image acquisition module, used for acquiring an image to be detected; A detection module, used to detect the image to be detected using a target detection model to obtain spatial position detection information of the target object; Wherein, the target detection model is trained according to the model training method according to claim 7.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the sample creation method as described in any one of claims 1 to 6; or, when the processor executes the program, it implements the model training method as described in claim 7; or, when the processor executes the program, it implements the target detection method as described in claim 8.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the sample creation method as described in any one of claims 1 to 6; or, when the program is executed by a processor, it implements the model training method as described in claim 7; or, when the program is executed by a processor, it implements the target detection method as described in claim 8.

Images