RGBD image combined negative sample image generation method, system and device and medium

Abstract

The invention discloses a combined negative sample image generation method for RGBD images. The method comprises the following steps: S1, obtaining a first RGB image and a first depth map; s2, identifying that a core region of the target object and a region outside the core region on the first RGB image and the first depth map are non-core regions, and combining and splicing the first RGB image and the first depth map to obtain a composite image; s3, erasing or filling the core region of the composite image, and segmenting the composite image into a second RGB image and a second depth map to obtain a first negative sample image set; and S4, operating the non-core region of the composite image, and generating a third RGB image and a third depth map by using at least part of image content of the non-core region to obtain a second negative sample image set. When the generated image is used for model training, the error passing rate of the model can be effectively reduced.

Description

Background Art

[0002] In image recognition, training samples are of great significance to the recognition model. The number of training sample sets is very important for the learning of deep learning models. A sufficient number of training sample sets is a prerequisite for a deep learning model to be able to effectively recognize. However, collecting a large number of images for training through equipment leads to excessive costs.

[0003] Data augmentation is one of the commonly used techniques in deep learning. It is mainly used to increase the training data set and make the data set as diverse as possible, so that the trained model has stronger generalization ability. In practical applications, different augmentation methods are suitable for different types. At present, data augmentation mainly includes: horizontal / vertical flipping, rotation, scaling, cropping, shearing, translation, contrast, color jitter, noise, etc. However, the current data augmentation scheme is mainly used to increase the positive samples of the training sample set, thereby improving the pass rate of model recognition, but lacks an augmentation method for negative samples to reduce the error pass rate of the model. The current data augmentation scheme is also prone to cause the model's error recognition rate to be too high, which needs to be corrected by negative samples.

[0004] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0005] To this end, the present invention combines the first RGB image and the first depth map, identifies the core area of ​​the target object, performs various types of processing on the target object and the background, and generates multiple types of negative samples, which can effectively reduce the error pass rate of the model when used for model training.

[0006] In a first aspect, the present invention provides a method for generating a combined negative sample image of an RGBD image, characterized in that it comprises the following steps:

[0007] Step S1: Acquire a first RGB image and a first depth map; wherein the first RGB image and the first depth map are aligned and both contain a target object and a background;

[0008] Step S2: identifying a core area of ​​the target object, wherein the area outside the core area on the first RGB image and the first depth map is a non-core area, and the first RGB image and the first depth map are combined and spliced ​​to obtain a composite image;

[0009] Step S3: performing an erasing or filling operation on the core area of ​​the composite image, and then dividing the composite image into a second RGB image and a second depth map to obtain a first negative sample image set;

[0010] Step S4: operating the non-core area of ​​the composite image, and using at least part of the image content of the non-core area to generate a third RGB image and a third depth map, respectively, to obtain a second negative sample image set.

[0011] Optionally, the method for generating a combined negative sample image of an RGBD image is characterized in that step S2 comprises:

[0012] Step S21: selecting a corresponding core area recognition model according to the type of the target object;

[0013] Step S22: if the target object is of the first category, the core area recognition model recognizes a specific portion of the target object as a core area;

[0014] Step S23: if the target object is of the second category, the core area recognition model recognizes a fixed percentage of the target object as the core area;

[0015] Step S24: combining and splicing the first RGB image and the first depth map to obtain a composite image.

[0016] Optionally, the method for generating a combined negative sample image of an RGBD image is characterized in that, in step S21, the target object is classified according to the distribution of feature points of the target object.

[0017] Optionally, the combined negative sample image generation method for an RGBD image is characterized in that if the feature points are concentrated, the target object is classified into the first category; if the feature points are dispersed, the target object is classified into the second category; if the feature points are unevenly distributed and sparse, the area consisting of fixed pixels extending outward from the feature points is taken as the core area and marked as the first category.

[0018] Optionally, the method for generating a combined negative sample image of an RGBD image is characterized in that when operating the first RGB image or the first depth map, the operation method for the first RGB image or the first depth map may be different.

[0019] Optionally, the method for generating a combined negative sample image of an RGBD image is characterized in that step S3 comprises:

[0020] Step S31: setting all pixels in the core area to the same constant to generate a negative sample image set;

[0021] Step S32: sampling from the non-core area, replacing all pixels in the core area, and generating a negative sample image set;

[0022] Step S33: sampling from the candidate images, replacing all pixels in the core area, and generating a negative sample image set; wherein the candidate images do not contain the target object;

[0023] Step S34: Segment the negative sample image set into a second RGB image and a second depth map to obtain a first negative sample image set.

[0024] Optionally, the method for generating a combined negative sample image of an RGBD image is characterized in that step S4 comprises:

[0025] Step S41: randomly sampling the non-core area through a collection frame, and generating a negative sample image set;

[0026] Step S42: setting some pixels in the non-core area to constants, and then randomly sampling the non-core area through a collection frame to generate a negative sample image set;

[0027] Step S43: sampling from the candidate image, replacing some pixels in the non-core area, and then randomly sampling the non-core area through a collection frame to generate a negative sample image set; wherein the candidate image does not contain the target object;

[0028] Step S44: Segment the negative sample image set into a third RGB image and a third depth map to obtain a second negative sample image set.

[0029] In a second aspect, the present invention provides a system for generating a combined negative sample image of an RGBD image, which is used to implement any of the above-mentioned methods for generating a combined negative sample image of an RGBD image, and is characterized by comprising:

[0030] An acquisition module, configured to acquire a first RGB image and a first depth map; wherein the first RGB image and the first depth map are aligned and both contain a target object and a background;

[0031] A core area module, used for identifying the core area of ​​the target object, the area outside the core area on the first RGB image and the first depth map is a non-core area, and the first RGB image and the first depth map are combined and spliced ​​to obtain a composite image;

[0032] A first negative sample module is used to erase or fill the core area of ​​the composite image, and then divide the composite image into a second RGB image and a second depth map to obtain a first negative sample image set;

[0033] The second negative sample module is used to operate the non-core area of ​​the composite image, and use at least part of the image content of the non-core area to generate a third RGB image and a third depth map, respectively, to obtain a second negative sample image set.

[0034] In a third aspect, the present invention provides a combined negative sample image generation device for an RGBD image, characterized in that it includes:

[0035] processor;

[0036] a memory storing executable instructions of the processor;

[0037] The processor is configured to execute any one of the steps of the above-mentioned method for generating a combined negative sample image of an RGBD image by executing the executable instructions.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any one of the aforementioned methods for generating a combined negative sample image of an RGBD image are implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention obtains negative samples through positive samples, and can correct the judgment of the model together with the positive samples during model training, which can greatly reduce the algorithm's misrecognition of positive samples, thereby reducing the false pass rate and improving accuracy.

[0041] The present invention obtains various types of negative samples by identifying the core area of ​​the target object and operating the core area and non-core area respectively. Through a variety of different types of negative samples, various types of erroneous recognition of the model can be effectively optimized, and has wide applicability.

[0042] The present invention processes the combination of the first RGB image and the first depth map, and can quickly generate RGBD images in batches with high efficiency, and the generated negative samples will not interfere with the normal recognition of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0044] Figure 1 This is a flowchart of a method for generating a combined negative sample image of an RGBD image in an embodiment of the present invention;

[0045] Figure 2 is a schematic diagram of a first negative sample image in an embodiment of the present invention;

[0046] Figure 3 is a schematic diagram of a second negative sample image in an embodiment of the present invention;

[0047] Figure 4 A flowchart of a step of identifying a core area in an embodiment of the present invention;

[0048] Figure 5 This is a flow chart of steps for generating a first negative sample image set in an embodiment of the present invention;

[0049] Figure 6 This is a flow chart of steps for generating a second negative sample image set in an embodiment of the present invention;

[0050] Figure 7 It is a structural schematic diagram of a combined negative sample image generation system for RGBD images in an embodiment of the present invention;

[0051] Figure 8 is a structural schematic diagram of a combined negative sample image generating device for an RGBD image in an embodiment of the present invention; and

[0052] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] An embodiment of the present invention provides a method for generating a combined negative sample image of an RGBD image, aiming to solve the problems existing in the prior art.

[0056] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0057] The present invention combines the first RGB image and the first depth map, identifies the core area of ​​the target object, performs various types of processing on the target object and the background, and generates multiple types of negative samples. When used for model training, the error pass rate of the model can be effectively reduced.

[0058] Figure 1 FIG. 1 is a flowchart of a method for generating a combined negative sample image of an RGBD image in an embodiment of the present invention. Figure 1 As shown, the steps of a method for generating a combined negative sample image of an RGBD image in an embodiment of the present invention include:

[0059] Step S1: Acquire a first RGB image and a first depth map.

[0060] In this step, the first RGB image and the first depth map are aligned, and both contain a target object and a background. The target object is an object that the model needs to recognize, such as a cat, a dog, a face, a palm, a human body, a house, etc. Different models have different recognition objects. This specification is collectively referred to as a target object. The model targeted by this embodiment is a deep learning network model. The deep learning network model can identify similar target objects by learning positive samples and negative samples, thereby achieving rapid discovery. The first RGB image and the first depth map in this embodiment are positive samples. It should be noted that although the present invention is described in terms of RGB images and depth maps, those skilled in the art can understand that the present invention is also applicable to combinations of other images and depth maps other than RGB images, such as infrared images and depth maps, speckle images and depth maps, etc. Therefore, combinations of other images and depth maps also fall within the scope of protection of the present invention.

[0061] Step S2: Identify the core area of ​​the target object, the area outside the core area on the first RGB image and the first depth map is a non-core area, and combine and splice the first RGB image and the first depth map to obtain a composite image.

[0062] In this step, part of the target object is taken as the core area, and the area outside the core area in the background and the target object is taken as the non-core area. The core area is the area with the most obvious features in the target object, and the existence of the target object can be determined by these features. The core area exists in both the first RGB image and the first depth map. Faces, hands, etc. are core areas. The existence of the human body can be determined by faces and hands. Other areas except the core area are non-core areas. Due to the different information of the first RGB image and the first depth map, the initial core areas identified in the two images are different. When identifying the core area, the first RGB image and the first depth map are used as the identification objects, and the initial core areas obtained on the first RGB image and the first depth map are merged, that is, the initial core areas are operated by "or" operation to obtain the core area. When operating the first RGB image and the first depth map, the core area is used as the operation basis. Compared with the first RGB image or the first depth map alone, the core area obtained by the scheme of jointly processing the first RGB image and the first depth map is larger and more comprehensive. The first RGB image and the first depth map are combined and spliced ​​to obtain a synthetic image. When performing combined stitching, the first RGB image and the first depth map may be stitched horizontally or vertically. The order of the first RGB image and the first depth map may be swapped arbitrarily. In some embodiments, multiple groups of first RGB images and first depth maps are combined and stitched.

[0063] Step S3: performing an erasing or filling operation on the core area of ​​the composite image, and then dividing the composite image into a second RGB image and a second depth map to obtain a first negative sample image set.

[0064] In this step, the erasing operation is to set the pixel value to 0. In the specific operation, the corresponding operation is performed according to the type of image. For example, in the corresponding part of the first depth map, the pixel points are directly set to 0; in the corresponding part of the first RGB image, the pixel points are directly set to (0,0,0). The filling operation is to cover the pixel values ​​of the core area and modify them to other pixel values. The values ​​of multiple pixels can be different or the same. The first negative sample image set is an image set formed by removing the area with the most obvious features of the target object. The size of the first negative sample image is the same as the size of the first RGB image and the first depth map, respectively. Figure 2 As shown, the features of the target object in the first negative sample image are not obvious, and the existence of the target object cannot be clearly determined. After this step is completed, the synthesized image is segmented to obtain a second RGB image and a second depth map, which constitute the first negative sample image set. When the synthesized image is segmented, the order of synthesis in step S2 is opposite, and the image is segmented into an RGB image and a depth map according to the format.

[0065] Step S4: operating the non-core area of ​​the composite image, and using at least part of the image content of the non-core area to generate a third RGB image and a third depth map, respectively, to obtain a second negative sample image set.

[0066] In this step, operations are performed on the area outside the core area, including full or partial horizontal / vertical flipping, rotation, scaling, cropping, shearing, translation, contrast, color dithering, noise, etc. During the operation, operations can be performed on a part of the non-core area or on multiple parts of the non-core area. If operations are performed on multiple parts of the non-core area, the operations on the multiple parts can be different.

[0067] When generating the second negative sample image, the non-core area may be used for sampling to generate the second negative sample image. Since the image size is reduced due to the sampling of part of the image, the image may be expanded to make the image size the same as the first RGB image and the first depth map, such as by inserting pixels at adjacent pixels through interpolation to increase the image size; or by using image stitching and other solutions to make the image size the same as the first RGB image and the first depth map. Figure 3As shown, the second negative sample image is part of the non-core area. When operating the composite image, the operation method for the first RGB image part or the first depth map part may be different. After this step is completed, the composite image is segmented to obtain a third RGB image and a third depth map to form a second negative sample image set. When segmenting the composite image, the image is segmented into an RGB image and a depth map according to the format, in the opposite order of synthesis in step S2.

[0068] In some embodiments, the operation of step S3 is performed on the core area, and then the operation of step S4 is performed to generate a second negative sample image set. The second negative sample image is generated using part of the non-core area and at least part of the processed core area. Since the core area has been processed to be unable to effectively identify the target object, various operations on the processed image cannot effectively obtain the target object, but it has the characteristics of the non-core area and can be well used as a negative sample.

[0069] Figure 4 FIG. 1 is a flowchart of a step of identifying a core area in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of identifying a core area includes:

[0070] Step S21: selecting a corresponding core area recognition model according to the type of the target object.

[0071] In this step, different target objects have different characteristics, so the corresponding core area recognition model can be selected according to the type of the target object.

[0072] In some embodiments, the target object is identified by a classification model, and a core area recognition model is assigned by a preset correspondence. A core area recognition model is preset for each target object type. For example, if the target object is identified as a human body, it is identified according to a preset core area recognition model corresponding to the human body; if the target object is identified as a chair, it is identified according to a preset core area recognition model corresponding to the chair.

[0073] In some embodiments, the target object is classified according to the distribution of the feature points of the target object. The feature points of the target object are identified by the target recognition model, and then the target object is classified according to the distribution of the feature points. The distribution of the feature points can well describe the key features of the target object, and thus can be used to evaluate the position of the core area of ​​the target object. If the feature points are concentrated, the target object is classified into the first category; if the feature points are dispersed, the target object is classified into the second category. If the feature points are unevenly distributed and sparse, the area consisting of fixed pixels extending outward from the feature points is taken as the core area, and it is marked as the first category.

[0074] Step S22: If the target object is of the first category, the core area recognition model recognizes a specific portion of the target object as a core area.

[0075] In this step, the first category is the target objects with obvious features, such as the left eye, right eye, nose, mouth, ears, eyebrows, etc. For the face, these key feature areas are taken as the core area. Similarly, for other target objects with obvious features, the key parts are taken as the core area.

[0076] Step S23: If the target object is of the second category, the core area recognition model recognizes a fixed percentage of the target object as the core area.

[0077] In this step, the second category is characterized by the whole, and it is impossible to identify the target object based on the local features, such as tables, chairs, etc. It is impossible to identify a table simply by the legs, but it may also be a chair, a pole, or other vertical objects. At this time, a fixed percentage of the target object is used as the core area. The fixed percentage is between 70% and 100%. The core area is a continuous whole.

[0078] Step S24: combining and splicing the first RGB image and the first depth map to obtain a composite image.

[0079] In this step, since the first RGB image and the first depth map have the same size, they can be directly stitched horizontally or vertically. The size after stitching is an integer multiple of the size of the first RGB image or the first depth map. The composite image can combine multiple images together, which is more efficient when processed.

[0080] This embodiment identifies specific parts of content or fixed percentages of areas as core areas according to different target object types. The best method can be selected according to the target object type to minimize the scope of the core area, expand the processing scope of the image, and minimize the error pass rate of the model.

[0081] Figure 5 FIG. 1 is a flow chart of steps for generating a first negative sample image set in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of generating a first negative sample image set includes:

[0082] Step S31: setting all pixels in the core area to the same constant to generate a negative sample image set.

[0083] In this step, constants are used to assign fixed values ​​to pixels. The type of constant is the same as the type of pixel. For example, if the positive sample is a depth map and the pixel is a numerical value, the constant is a numerical value; if the positive sample is an RGB map and the pixel is three numbers (x, y, z), the constant is also three numbers (x', y', z'). The constant can be either 0 or any number greater than 0.

[0084] Step S32: sampling from the non-core area, replacing all pixels in the core area, and generating a negative sample image set.

[0085] In this step, compared with the previous step, this step replaces the pixel points in the core area with the pixel points in the non-core area, so that the content of the non-core area occupies all areas of the image, thereby eliminating the content of the core area and obtaining a negative sample image set with only the non-core area. When sampling, for the first RGB image, only the non-core area of ​​the first RGB image is sampled; for the first depth map, only the non-core area of ​​the first depth map is sampled. The sampling range and size for the first RGB image and the first depth map can be different.

[0086] Step S33: sampling from the candidate images, replacing all pixels in the core area, and generating a negative sample image set.

[0087] In this step, the candidate images do not contain the target object. The candidate images are not positive sample sets, but additional images that are not associated with the target object, such as landscape images without the target object, city images without the target object, abstract images without the target object, etc. The candidate images are also RGB images and depth images, but they can be RGB images and depth images that do not correspond one to one, or even completely unrelated RGB images and depth images.

[0088] Step S34: Segment the negative sample image set into a second RGB image and a second depth map to obtain a first negative sample image set.

[0089] In this step, the negative sample image set generated in the previous step is segmented according to the synthesis line to obtain one-to-one corresponding RGB images and depth maps to form the first negative sample image set. For example, if the first RGB image and the first depth map are combined along the edge AB of the first RGB image, then segmentation is performed along the edge AB in this step to obtain the segmented second RGB image and the second depth map. Multiple groups of second RGB images and second depth maps constitute the first negative sample image set.

[0090] This embodiment replaces the core area with different contents to generate negative sample images of various types and styles, which greatly enriches the types of negative sample images, helps to reduce the false pass rate of the model, and plays a role in reducing costs and increasing efficiency.

[0091] Figure 6 FIG. 1 is a flow chart of steps for generating a second negative sample image set in an embodiment of the present invention. Figure 6 As shown, in an embodiment of the present invention, a step of generating a second negative sample image set includes:

[0092] Step S41: randomly sampling the non-core area through a collection frame, and generating a negative sample image set.

[0093] In this step, the acquisition frame acquires part of the content in the non-core area and generates it as a negative sample image. The acquisition frame randomly acquires multiple times to generate a set of negative sample images. The negative sample image has the same size as the first RGB image and the first depth map. This step also performs a size transformation on the content acquired by the acquisition frame, such as interpolation to make the size the same as the first RGB image and the first depth map. When sampling, for the first RGB image, only the non-core area of ​​the first RGB image is sampled; for the first depth map, only the non-core area of ​​the first depth map is sampled. The sampling range and size for the first RGB image and the first depth map may be different.

[0094] Step S42: setting some pixels in the non-core area to constants, and then randomly sampling the non-core area through a collection frame to generate a negative sample image set.

[0095] In this step, some pixels in the non-core area are set to constants. The constant here refers to a fixed value, not to the same pixel value for all reset pixels. For example, some non-core areas can be set to rainbow or gradient shapes. The constant in this step refers to a value that makes the reset area have no interaction with other areas and presents a certain regularity. The constant can change regularly with the change of position on the image.

[0096] Step S43: sampling from the candidate images, replacing some pixels in the non-core area, and then randomly sampling the non-core area through a collection frame to generate a negative sample image set.

[0097] In this step, the candidate image does not contain the target object. The candidate image is also an RGB image and a depth map, but can be an RGB image and a depth map that do not correspond one to one, or can even be an RGB image and a depth map that have no correlation.

[0098] Step S44: Segment the negative sample image set into a third RGB image and a third depth map to obtain a second negative sample image set.

[0099] In this step, the negative sample image set generated in the previous step is segmented according to the synthesis line to obtain one-to-one corresponding RGB images and depth maps to form the first negative sample image set. For example, if the first RGB image and the first depth map are combined along the edge AB of the first RGB image, then segmentation is performed along the edge AB in this step to obtain the segmented third RGB image and the third depth map. Multiple groups of third RGB images and third depth maps constitute the second negative sample image set.

[0100] This embodiment replaces all or part of the non-core area in a variety of ways, and generates negative sample image sets using three methods: constants, non-core area content, and alternative images, enriching the types of negative sample images, which is beneficial to reducing the false pass rate of the model and playing a role in reducing costs and increasing efficiency.

[0101] Figure 7 FIG. 1 is a schematic diagram of the structure of a system for generating a combined negative sample image of an RGBD image in an embodiment of the present invention. Figure 7 As shown, a combined negative sample image generation system for RGBD images in an embodiment of the present invention includes:

[0102] An acquisition module, configured to acquire a first RGB image and a first depth map; wherein the first RGB image and the first depth map are aligned and both contain a target object and a background;

[0103] A core area module, used for identifying the core area of ​​the target object, the area outside the core area on the first RGB image and the first depth map is a non-core area, and the first RGB image and the first depth map are combined and spliced ​​to obtain a composite image;

[0104] A first negative sample module is used to erase or fill the core area of ​​the composite image, and then divide the composite image into a second RGB image and a second depth map to obtain a first negative sample image set;

[0105] The second negative sample module is used to operate the non-core area of ​​the composite image, and use at least part of the image content of the non-core area to generate a third RGB image and a third depth map, respectively, to obtain a second negative sample image set.

[0106] Specifically, the acquisition module obtains the aligned first RGB image and the first depth map. The model can identify the first RGB image and the first depth map to obtain the target object. The core area module identifies the core area of ​​the target object, that is, the key features of the target object. The first negative sample module erases or fills the core area to generate a first negative sample image set. The second negative sample module generates a second negative sample set by operating on the non-core area, such as sampling.

[0107] In some embodiments, the second negative sample module further processes the images processed by the first negative sample module to generate a second negative sample image set.

[0108] This embodiment combines the first RGB image and the first depth map to generate multiple types of negative samples by identifying the core area of ​​the target object and performing various types of processing on the target object and the background. When used for model training, the error pass rate of the model can be effectively reduced.

[0109] An embodiment of the present invention further provides a combined negative sample image generation device for an RGBD image, comprising a processor and a memory in which executable instructions of the processor are stored. The processor is configured to execute the steps of a combined negative sample image generation method for an RGBD image by executing the executable instructions.

[0110] As described above, this embodiment combines the first RGB image and the first depth map, identifies the core area of ​​the target object, performs various types of processing on the target object and the background, and generates multiple types of negative samples. When used for model training, the error pass rate of the model can be effectively reduced.

[0111] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0112] Figure 8 Schematic diagram of the structure of a combined negative sample image generation device for RGBD images in an embodiment of the present invention. Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0113] like Figure 8 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0114] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned method for generating a combined negative sample image of an RGBD image. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0115] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0116] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0117] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0118] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 8 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0119] In an embodiment of the present invention, a computer-readable storage medium is further provided for storing a program, and when the program is executed, the steps of a method for generating a combined negative sample image of an RGBD image are implemented. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above-mentioned method for generating a combined negative sample image of an RGBD image in this specification.

[0120] As shown above, this embodiment combines the first RGB image and the first depth map, identifies the core area of ​​the target object, performs various types of processing on the target object and the background, and generates multiple types of negative samples, which can effectively reduce the error pass rate of the model when used for model training.

[0121] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig. 9 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0122] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0123] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0124] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0125] This embodiment combines the first RGB image and the first depth map to generate multiple types of negative samples by identifying the core area of ​​the target object and performing various types of processing on the target object and the background. When used for model training, the error pass rate of the model can be effectively reduced.

[0126] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0127] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A combined negative sample image generation method for RGBD images, It is characterized in that The steps include: Step S1: Acquire a first RGB image and a first depth map; wherein the first RGB image and the first depth map are aligned and both contain a target object and a background; Step S2: identifying a core area of ​​the target object, wherein the area outside the core area on the first RGB image and the first depth map is a non-core area, and the first RGB image and the first depth map are combined and spliced ​​to obtain a composite image; Step S3: performing an erasing or filling operation on the core area of ​​the composite image, and then dividing the composite image into a second RGB image and a second depth map to obtain a first negative sample image set; Step S4: operating the non-core area of ​​the composite image, and using at least part of the image content of the non-core area to generate a third RGB image and a third depth map, respectively, to obtain a second negative sample image set.

2. The method for generating a combined negative sample image of an RGBD image according to claim 1, It is characterized in that Step S2 includes: Step S21: selecting a corresponding core area recognition model according to the type of the target object; Step S22: if the target object is of the first category, the core area recognition model recognizes a specific portion of the target object as a core area; Step S23: if the target object is of the second category, the core area recognition model recognizes a fixed percentage of the target object as the core area; Step S24: combining and splicing the first RGB image and the first depth map to obtain a composite image.

3. The method for generating a combined negative sample image of an RGBD image according to claim 2, It is characterized in that In step S21, the target object is classified according to the distribution of the feature points of the target object.

4. The method for generating a combined negative sample image of an RGBD image according to claim 3, It is characterized in that If the feature points are concentrated, the target object is classified into the first category; if the feature points are dispersed, the target object is classified into the second category; if the feature points are unevenly distributed and sparse, the area consisting of fixed pixels extending outward from the feature points is taken as the core area and marked as the first category.

5. The method for generating a combined negative sample image of an RGBD image according to claim 1, It is characterized in that When operating the first RGB image or the first depth map, the operating methods of the first RGB image or the first depth map may be different.

6. The method for generating a combined negative sample image of an RGBD image according to claim 1, It is characterized in that Step S3 includes: Step S31: setting all pixels in the core area to the same constant to generate a negative sample image set; Step S32: sampling from the non-core area, replacing all pixels in the core area, and generating a negative sample image set; Step S33: sampling from the candidate images, replacing all pixels in the core area, and generating a negative sample image set; wherein the candidate images do not contain the target object; Step S34: Segment the negative sample image set into a second RGB image and a second depth map to obtain a first negative sample image set.

7. The method for generating a combined negative sample image of an RGBD image according to claim 1, It is characterized in that The step S4 comprises: Step S41: randomly sampling the non-core area through a collection frame, and generating a negative sample image set; Step S42: setting some pixels in the non-core area to constants, and then randomly sampling the non-core area through a collection frame to generate a negative sample image set; Step S43: sampling from the candidate image, replacing some pixels in the non-core area, and then randomly sampling the non-core area through a collection frame to generate a negative sample image set; wherein the candidate image does not contain the target object; Step S44: Segment the negative sample image set into a third RGB image and a third depth map to obtain a second negative sample image set.

8. A system for generating a combined negative sample image of an RGBD image, for implementing the method for generating a combined negative sample image of an RGBD image according to any one of claims 1 to 7, It is characterized in that include: An acquisition module, configured to acquire a first RGB image and a first depth map; wherein the first RGB image and the first depth map are aligned and both contain a target object and a background; A core area module, used for identifying the core area of ​​the target object, the area outside the core area on the first RGB image and the first depth map is a non-core area, and the first RGB image and the first depth map are combined and spliced ​​to obtain a composite image; A first negative sample module is used to erase or fill the core area of ​​the composite image, and then divide the composite image into a second RGB image and a second depth map to obtain a first negative sample image set; The second negative sample module is used to operate the non-core area of ​​the composite image, and use at least part of the image content of the non-core area to generate a third RGB image and a third depth map, respectively, to obtain a second negative sample image set.

9. A combined negative sample image generation device for RGBD images, It is characterized in that include: processor; a memory storing executable instructions of the processor; The processor is configured to perform the steps of the combined negative sample image generation method for RGBD images according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, It is characterized in that When the program is executed, the steps of the method for generating a combined negative sample image of an RGBD image according to any one of claims 1 to 7 are implemented.

Images