Image processing method and device, electronic equipment and storage medium

Abstract

The invention provides an image processing method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. Comprising the following steps: extracting at least one foreground object image from each acquired initial image; determining scene depth information corresponding to each scene background image according to each collected scene background image; determining a target area in each scene background image according to each scene background image; and according to the scene depth information corresponding to each scene background image and the target area in each scene background image, embedding the at least one foreground object image into the at least one scene background image to generate a plurality of composite images. According to the method, expensive manual data annotation is replaced by a synthetic image generation mode, accurate label information can be provided for training of the downstream detection model due to the fact that the foreground object in the synthetic image has a clear identifier, and the training accuracy of the downstream detection model can be improved on the premise that time and labor consumed by manual annotation are reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] Since images of special traffic scenes are difficult to collect and annotate, the trained downstream object detection models do not perform well in detection tasks. Therefore, it is particularly important to generate high-quality traffic scene content images to provide training data for downstream model training.

[0003] Currently, most training sample data are obtained by collecting existing traffic scene background images in the database and manually annotating them.

[0004] However, the labeling process relies heavily on manual labor, which is costly and has poor accuracy, thus affecting the detection effect of downstream models. Summary of the invention

[0005] The purpose of this application is to provide an image processing method, device, electronic device and storage medium to address the deficiencies in the above-mentioned prior art, thereby replacing manual annotation with synthetic images, reducing manpower consumption and improving the accuracy of downstream model detection.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows: In a first aspect, an embodiment of the present application provides an image processing method, comprising: Extracting at least one foreground object image from each acquired initial image; Determine the scene depth information corresponding to each scene background image according to the collected background images of each scene; Determining a target area in each scene background image according to each scene background image; At least one foreground object image is embedded into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate a plurality of composite images.

[0007] Optionally, extracting at least one foreground object image from each acquired initial image includes: Collecting a plurality of initial images, each of which contains at least one foreground object; According to each initial image and at least one foreground object description information corresponding to each initial image, a large language visual model is used to extract each foreground object image corresponding to each foreground object description information from each initial image.

[0008] Optionally, extracting foreground object images corresponding to the foreground object description information from the initial images using a large language visual model according to the initial images and the at least one foreground object description information corresponding to the initial images includes: The large language visual model encodes the initial image and the target foreground object description information corresponding to the initial image to generate fused encoding information; Segmenting and decoding the fused coding information, extracting a binary mask image of the target foreground object corresponding to the target foreground object description information from the initial image, and converting the binary mask image to obtain a bounding box of the target foreground object; According to the bounding box of the target foreground object, a target foreground object image is extracted.

[0009] Optionally, determining scene depth information corresponding to each scene background image according to each scene background image collected includes: A depth estimation model and a segmentation model are respectively used to encode the scene background image to obtain a feature map of the scene background image; Decoding the feature map of the scene background image to obtain a depth map of the scene background image; Determine depth information of the scene background image according to the depth map.

[0010] Optionally, determining the target area in each scene background image according to each scene background image includes: Generate a road surface mask image in the scene background image according to the scene background image; Determining an embeddable area in the scene background image according to the road mask image; Determine region ratio information according to the boundary information of the embeddable region and the width information of the scene background image; the region ratio information is used to characterize the ratio of the target region to be selected to the embeddable region; The region segmentation position is determined according to the region ratio information and the boundary information of the embeddable region, and the embeddable region is segmented based on the region segmentation position to obtain the target region.

[0011] Optionally, determining the embeddable area in the scene background image according to the road mask image includes: Determining, according to the road mask image, a first distance of the road surface close to the bottom edge of the scene background image and a second distance of the road surface close to the top edge of the scene background image; Determining a first boundary line according to the first distance and a boundary of the road surface mask image; determining a second boundary line according to the second distance and a boundary of the road surface mask image; Determine a third boundary line and a fourth boundary line respectively according to the first boundary line, the second boundary line and the image area of ​​the road surface mask image; The embeddable area is formed by the first boundary line, the second boundary line, the third boundary line, and the fourth boundary line.

[0012] Optionally, determining the area ratio information according to the boundary information of the embeddable area and the width information of the scene background image includes: The area ratio information is determined according to the endpoint coordinate information of the first boundary line of the embeddable area, the endpoint coordinate information of the second boundary line, and the width information of the scene background image.

[0013] Optionally, before determining the region segmentation position according to the region ratio information and the boundary information of the embeddable region, and segmenting the embeddable region based on the region segmentation position to obtain the target region, the method further includes: If the area ratio information is greater than a preset threshold, the area ratio information is adjusted to obtain adjusted area ratio information.

[0014] Optionally, determining the region segmentation position according to the region ratio information and the boundary information of the embeddable region includes: The area division position is determined according to the endpoint coordinate information of the first boundary line of the embeddable area, the endpoint coordinate information of the second boundary line, and the area ratio information.

[0015] Optionally, embedding at least one foreground object image into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate a plurality of composite images includes: Determine a target embedding position from a target area in a target scene background image; the target embedding position is any position in the target area; Determining the depth information of the target embedding position according to the scene depth information corresponding to the target scene background image; Determine the image size of the target foreground object to be embedded at the target embedding position in the target scene background image according to the depth information of the target embedding position and the mapping relationship between the depth information and the image embedding size; According to the image size, the target foreground object image is embedded in the target embedding position in the target scene background image to generate a composite image.

[0016] Optionally, embedding at least one foreground object image into at least one scene background image to generate a plurality of composite images comprises: embedding one or more of the foreground object images into a current scene background image to generate at least one synthetic image corresponding to the current scene background image, wherein the current scene background image is any scene background image; The multiple composite images are obtained according to at least one composite image corresponding to each scene background image.

[0017] Optionally, embedding at least one foreground object image into at least one scene background image to generate a plurality of composite images comprises: If the number of foreground object images currently embedded in the target scene background image does not exceed the preset upper limit of the embedding number, the foreground object image to be embedded is embedded into the target scene background image according to the embedding position of each currently embedded foreground object image to generate a current composite image.

[0018] Optionally, after generating a plurality of composite images, the method further includes: The pre-trained image harmonization model is used to repair the inharmonious areas in each composite image to obtain an optimized composite image.

[0019] Optionally, the training process of the image harmonization model includes: Collecting a training sample data set, the training sample data set comprising: a plurality of groups of sample data, each group of sample data comprising: a real shot image and a synthetic image constructed based on the real shot image; The image harmonization model is obtained by training using the training sample data set.

[0020] In a second aspect, an embodiment of the present application further provides an image processing device, including: an acquisition module, a determination module, and an embedding module; The acquisition module is used to extract at least one foreground object image from each acquired initial image; The determination module is used to determine the scene depth information corresponding to each scene background image according to the collected scene background images; The determination module is used to determine the target area in each scene background image according to each scene background image; The embedding module is used to embed at least one foreground object image into at least one scene background image according to the scene depth information corresponding to each scene background image and the target area in each scene background image to generate multiple composite images.

[0021] Optionally, the acquisition module is specifically used to collect a plurality of initial images, each of which contains at least one foreground object; According to each initial image and at least one foreground object description information corresponding to each initial image, a large language visual model is used to extract each foreground object image corresponding to each foreground object description information from each initial image.

[0022] Optionally, the acquisition module is specifically used to encode the initial image and the target foreground object description information corresponding to the initial image respectively by the large language visual model to generate fused coding information; Segmenting and decoding the fused coding information, extracting a binary mask image of the target foreground object corresponding to the target foreground object description information from the initial image, and converting the binary mask image to obtain a bounding box of the target foreground object; According to the bounding box of the target foreground object, a target foreground object image is extracted.

[0023] Optionally, the determination module is specifically used to encode the scene background image using a depth estimation model and a segmentation model respectively to obtain a feature map of the scene background image; Decoding the feature map of the scene background image to obtain a depth map of the scene background image; Determine depth information of the scene background image according to the depth map.

[0024] Optionally, the determination module is specifically configured to generate a road surface mask image in the scene background image according to the scene background image; Determining an embeddable area in the scene background image according to the road mask image; Determine region ratio information according to the boundary information of the embeddable region and the width information of the scene background image; the region ratio information is used to characterize the ratio of the target region to be selected to the embeddable region; The region segmentation position is determined according to the region ratio information and the boundary information of the embeddable region, and the embeddable region is segmented based on the region segmentation position to obtain the target region.

[0025] Optionally, the determination module is specifically used to determine, according to the road surface mask image, a first distance of the road surface close to the bottom edge of the scene background image and a second distance of the road surface close to the top edge of the scene background image; Determining a first boundary line according to the first distance and a boundary of the road surface mask image; determining a second boundary line according to the second distance and a boundary of the road surface mask image; Determine a third boundary line and a fourth boundary line respectively according to the first boundary line, the second boundary line and the image area of ​​the road surface mask image; The embeddable area is formed by the first boundary line, the second boundary line, the third boundary line, and the fourth boundary line.

[0026] Optionally, the determination module is specifically configured to determine the area ratio information according to endpoint coordinate information of the first boundary line of the embeddable area, endpoint coordinate information of the second boundary line, and width information of the scene background image.

[0027] Optionally, an adjustment module is also included; The adjustment module is used to adjust the area ratio information if the area ratio information is greater than a preset threshold to obtain adjusted area ratio information.

[0028] Optionally, the determination module is specifically configured to determine the region segmentation position according to the endpoint coordinate information of the first boundary line of the embeddable region, the endpoint coordinate information of the second boundary line, and the region ratio information.

[0029] Optionally, the embedding module is specifically used to determine the target embedding position from the target area in the target scene background image; the target embedding position is any position in the target area; Determining the depth information of the target embedding position according to the scene depth information corresponding to the target scene background image; Determine the image size of the target foreground object to be embedded at the target embedding position in the target scene background image according to the depth information of the target embedding position and the mapping relationship between the depth information and the image embedding size; According to the image size, the target foreground object image is embedded in the target embedding position in the target scene background image to generate a composite image.

[0030] Optionally, the embedding module is specifically used to embed one or more of the foreground object images into a current scene background image to generate at least one synthetic image corresponding to the current scene background image, where the current scene background image is any scene background image; The multiple composite images are obtained according to at least one composite image corresponding to each scene background image.

[0031] Optionally, the embedding module is specifically used to embed the foreground object image to be embedded into the target scene background image according to the embedding position of each foreground object image currently embedded, if the number of foreground object images currently embedded in the target scene background image does not exceed a preset embedding number upper limit, so as to generate a current composite image.

[0032] Optionally, the adjustment module is further used to use a pre-trained image harmonization model to repair the inharmonious areas in each composite image to obtain an optimized composite image.

[0033] Optionally, it also includes: a training module; The training module is used to collect a training sample data set, wherein the training sample data set includes: a plurality of groups of sample data, each group of sample data includes: a real shot image and a synthetic image constructed based on the real shot image; The image harmonization model is obtained by training using the training sample data set.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the storage medium communicate through the bus, and the processor executes the machine-readable instructions to implement the image processing method provided in the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image processing method provided in the first aspect is executed.

[0036] The beneficial effects of this application are: The present application provides an image processing method, device, electronic device and storage medium, including: extracting at least one foreground object image from each acquired initial image; determining the scene depth information corresponding to each scene background image according to each acquired scene background image; determining the target area in each scene background image according to each scene background image; embedding at least one foreground object image into at least one scene background image according to the scene depth information corresponding to each scene background image and the target area in each scene background image to generate multiple synthetic images. Based on the foreground object image extracted from the initial image, combined with the depth information of the scene background image and the target area of ​​the scene background image, the method can determine the image size and embedding position of the foreground object image when embedded in the scene background image, so as to embed the foreground object image into the scene background image according to a reasonable size and position, and obtain a synthetic image in a traffic scene. The method realizes replacing expensive manual annotation data with the method of generating synthetic images. Since the foreground object in the synthetic image has a clear identification, it can provide accurate label information for the training of the downstream detection model. On the premise of reducing the time and labor of manual annotation, the training accuracy of the downstream detection model can also be improved.

[0037] In addition, extracting the foreground object image from the initial image based on the description information can improve the accuracy of the foreground object image extraction result and facilitate the generation of annotation information of the foreground object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the architecture of an image processing system provided in an embodiment of the present application; Figure 2 A flowchart of an image processing method provided in an embodiment of the present application; Figure 3 A flowchart of another image processing method provided in an embodiment of the present application; Figure 4 A schematic diagram of generating a binary image mask image of a foreground object provided in an embodiment of the present application; Figure 5 A flowchart of another image processing method provided in an embodiment of the present application; Figure 6 A flowchart of another image processing method provided in an embodiment of the present application; Figure 7 A flowchart of another image processing method provided in an embodiment of the present application; Figure 8 A flowchart of another image processing method provided in an embodiment of the present application; Fig. 9 A schematic diagram of a road surface mask of a scene background image provided in an embodiment of the present application; Fig.10 A schematic diagram of an embeddable area of ​​a scene background image provided in an embodiment of the present application; Fig.11 A flowchart of another image processing method provided in an embodiment of the present application; Fig.12 A flowchart of another image processing method provided in an embodiment of the present application; Fig.13 A schematic diagram of an image processing device provided in an embodiment of the present application; Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0041] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0042] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0043] Since images of special traffic scenes are difficult to collect and annotate, the performance quality of existing models in downstream detection tasks is generally low. Currently, most of the solutions in the market use multi-person annotation to train expert models, and then use the expert models to perform reasoning on large-scale data sets, and manually screen the reasoning results to obtain high-quality training data sets. Some other solutions directly use image fusion and generation technology to increase the number of training samples.

[0044] Existing image fusion technology introduces an instruction-based object addition pipeline, which can automatically insert objects into real scenes with reasonable sizes and positions. It consists of two parts: creating text instructions and fine-tuning the diffusion model to achieve reasonable generation. Specifically, first, by adopting object removal technology, instruction-based image pairs are created and a large background dataset is constructed. Then, a stable instruction-based diffusion model is trained on this dataset to learn diverse and reasonable object generation, avoiding reliance on costly manual annotations such as bounding boxes.

[0045] The shortcomings of the existing technology are as follows: 1. For the extraction of special categories of objects, ordinary text instructions will cause the object detection frame generated by the visual language model to be inaccurate, which directly affects the effect of image generation; 2. The elements in traffic scenes are usually complex, and more than one foreground element is generated in one picture. The perspective relationship between elements and elements, and between elements and background is particularly important. The existing solutions for generating images by fine-tuning the diffusion model often ignore the physical laws and are difficult to adapt to multi-scene deep modeling; 3. In the image synthesis process, the adaptation of lighting, shadow, perspective and background is crucial to the final synthesis effect. In traffic scenes, the targets to be detected are usually small objects. Therefore, traditional synthesis solutions such as Poisson fusion, Gaussian fusion and deep learning methods such as TF-ICON (Training-Free Image COmpositioN, a cross-domain image synthesis framework based on a diffusion model) and Blended (Blended Learning, hybrid learning mode) are too dependent on image gradients, resulting in small objects losing their original edge and color information after being integrated into dark backgrounds such as highway roads.

[0046] Based on this, this solution provides an image processing method, which extracts a foreground object image and extracts depth information of a scene background image, determines a target area in the scene background image based on a road mask image of the scene background image, and embeds the extracted foreground object image into the target area in the scene background image, wherein the image size of the foreground object image when embedded can be determined in combination with the depth information of the scene background image and the specific embedding position of the foreground object image in the target area, so that the foreground object image is embedded into the scene background image at a reasonable size and position to obtain a synthetic image in a traffic scene. It realizes replacing expensive manual annotation data with the method of generating synthetic images. Since the foreground object in the synthetic image has a clear identification, it can provide accurate label information for the training of the downstream detection model, and can also improve the training accuracy of the downstream detection model while reducing the time and labor of manual annotation.

[0047] Figure 1 A schematic diagram of the architecture of an image processing system provided in an embodiment of the present application; Figure 1As shown, based on the crawled initial image containing the foreground object, the initial image and the description information of the foreground object to be extracted are input into the large language vision model together, and the large language vision model obtains the fusion code by image encoding the initial image and text encoding the description information, and then the fusion code is segmented and decoded to obtain the foreground object image; similarly, the collected scene background image is processed based on the large language vision model to obtain the road mask image in the scene background image; in addition, the scene background image is processed based on the depth estimation model to extract the depth image corresponding to the scene background image, and the scene depth information of the scene background image can be obtained based on the depth image; thus, the foreground object image, the road mask image in the scene background image and the scene depth information are combined to perform image embedding processing to embed the foreground object image into a reasonable position in the scene background image, and obtain a synthetic image marked with the foreground object identifier, and the synthetic image can be used as training data for training the downstream detection model. Since the foreground object is a known object with a clear identifier, the synthetic image has accurate identifier information, which can improve the accuracy of the trained model.

[0048] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application; the execution subject of the method is a computer device such as a terminal or a server, such as Figure 2 As shown, the method may include: S101. Extract at least one foreground object image from each acquired initial image.

[0049] The initial image may be an image crawled from a website containing the foreground object to be detected. In a traffic scene, the foreground object to be detected may be an object of a special category that is not easily detected accurately. Assuming that the foreground object to be detected is a "roadside stone pier", the crawled initial image is an image containing various types of "roadside stone piers" in various scenes.

[0050] Of course, an initial image does not have to contain only one foreground object to be detected. Usually, the number and type of foreground objects contained in the initial image are not limited, and can be one or more.

[0051] Based on the crawled initial image, at least one foreground object image can be extracted from the initial image. When the initial image contains multiple different foreground objects, at least one foreground object image can be extracted from the initial image according to demand.

[0052] S102: Determine scene depth information corresponding to each scene background image according to the collected scene background images.

[0053] The scene background image can also be collected in advance, and can be collected from historical images stored in the camera. In a traffic scene, the scene background image can be an image containing a road scene.

[0054] Based on the collected scene background image, a scene depth distance relationship map corresponding to the scene background image can be estimated, which can also be called a scene depth map.

[0055] The scene depth map is a special form of image representation that records the distance information between each point in the scene and the observer (camera). In the depth map, the grayscale value or color value of each pixel represents the distance from the camera in the three-dimensional space corresponding to the point. This representation method allows the depth information to be presented intuitively in the form of an image. Therefore, the scene depth information can be obtained based on the scene depth map.

[0056] S103: Determine a target area in each scene background image according to each scene background image.

[0057] For the scene background image, the road mask image can be extracted from the scene background image, thereby determining the maximum bounding box of the road surface based on the road mask image. The maximum bounding box defines the boundary of the foreground object image when it is embedded in the scene background image, and the foreground object image is allowed to be embedded within the maximum bounding box.

[0058] However, in order to further improve the rationality and accuracy of the embedding position, a target area can be determined within the maximum bounding box through a preset algorithm, and the target area is a reasonable area that allows the foreground object image to be embedded.

[0059] S104: embed at least one foreground object image into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate a plurality of composite images.

[0060] In some embodiments, the embedding position of the foreground object image can be determined based on the target area in the scene background image, and the embedding size of the foreground object image can be determined based on the principle that objects appear larger when near and smaller when far away, so that the foreground object image is embedded in the embedding position in the scene background image according to the embedding size to generate a composite image.

[0061] Since the extracted foreground object images may include multiple and the scene background images may also include multiple, one or more foreground object images may be embedded in a scene background image, and the same foreground object image may also be embedded in different scene background images, thereby combining to generate multiple composite images, and the scene background or foreground object in each composite image is not completely consistent.

[0062] Optionally, the multiple synthetic images generated can be used as sample data for training downstream detection models, wherein when generating synthetic images, the foreground objects embedded in the scene background images have clear and accurate identification, so that the synthetic images have accurate labeling information and no manual labeling is required. The synthetic images can be directly used to train the model, and the trained model has better detection effects.

[0063] In summary, the image processing method provided by this embodiment includes: extracting at least one foreground object image from each acquired initial image; determining the scene depth information corresponding to each scene background image according to each acquired scene background image; determining the target area in each scene background image according to each scene background image; embedding at least one foreground object image into at least one scene background image according to the scene depth information corresponding to each scene background image and the target area in each scene background image to generate multiple synthetic images. Based on the foreground object image extracted from the initial image, combined with the depth information of the scene background image and the target area of ​​the scene background image, the method can determine the image size and embedding position of the foreground object image when embedded in the scene background image, so as to embed the foreground object image into the scene background image according to a reasonable size and position, and obtain a synthetic image in a traffic scene. It realizes replacing expensive manual annotation data with the method of generating synthetic images. Since the foreground object in the synthetic image has a clear identification, it can provide accurate label information for the training of the downstream detection model. On the premise of reducing the time and labor of manual annotation, the training accuracy of the downstream detection model can also be improved.

[0064] Figure 3 A flowchart of another image processing method provided in an embodiment of the present application; optionally, refer to Figure 3 As shown, in step S101, extracting at least one foreground object image from each acquired initial image may include: S201, collecting a plurality of initial images, each of which contains at least one foreground object.

[0065] In some embodiments, multiple initial images may be collected from a website. Based on the above description, it can be known that each initial image may include one or more foreground objects to be detected.

[0066] S202. According to each initial image and at least one foreground object description information corresponding to each initial image, a large language visual model is used to extract each foreground object image corresponding to each foreground object description information from each initial image.

[0067] According to the requirements, at least one foreground object can be extracted from each initial image. That is, one or more foreground objects can be extracted from an initial image.

[0068] According to the needs, it can be determined which foreground object you want to extract from the initial image, so as to generate the description information of the foreground object you want to extract. The foreground object description information and the initial image are input into the large language vision model together, and the foreground object image described by the foreground object description information can be extracted from the initial image.

[0069] For example, suppose that the foreground objects contained in the initial image are: a trash box and a puppy; if you want to extract the trash box in the initial image according to your needs, you can input the foreground object description information "trash box" and the initial image into the big language vision model together, and the big language vision model will extract the "trash box" image from the initial image based on the semantics described by the description information.

[0070] Figure 4 A schematic diagram of generating a binary mask image of a foreground object provided in an embodiment of the present application is provided. The foreground object description information and the initial image are used as inputs of a large language visual model, and the binary mask image of the foreground object described by the foreground object description information can be extracted after being processed by the large language visual model.

[0071] Figure 5 A flowchart of another image processing method provided in an embodiment of the present application; Figure 5 As shown, optionally, in step S202, based on each initial image and at least one foreground object description information corresponding to each initial image, using a large language visual model to extract each foreground object image corresponding to each foreground object description information from each initial image may include: S301, using a large language vision model to encode an initial image and target foreground object description information corresponding to the initial image to generate fused encoding information.

[0072] In one achievable manner, the large language vision model may perform image encoding on the initial image and text encoding on the target foreground object description information to obtain fused encoding information.

[0073] The target foreground object description information describes the target foreground object in the initial image, and the target foreground object may be any foreground object in the initial image.

[0074] S302 , segment and decode the fused coding information, extract a binary mask image of the target foreground object corresponding to the target foreground object description information from the initial image, and transform the binary mask image to obtain a bounding box of the target foreground object.

[0075] By segmenting and decoding the fused coding information, a binary mask image of the target foreground object described by the target foreground object description information can be extracted from the initial image, and the binary mask image can be converted into an external rectangular frame to obtain a bounding box of the target foreground object.

[0076] S303: Extract a target foreground object image according to the bounding box of the target foreground object.

[0077] Based on the bounding box of the target object, the image within the bounding box can be extracted to obtain the target foreground object image.

[0078] In another possible implementation, image processing techniques such as convolutional neural networks (CNNs) can be used to extract features from the input initial image. The extracted features include low-level features such as color, texture, shape, and more advanced semantic features.

[0079] The input target foreground object description information is then parsed to understand its semantic content, which usually involves natural language processing (NLP) technology, such as word segmentation, part-of-speech tagging, and syntactic analysis.

[0080] The parsed description information is matched with the extracted image features. By comparing the object features in the description information with the features in the image, the target foreground object image that matches the target foreground object description information is found.

[0081] Figure 6 A flowchart of another image processing method provided in an embodiment of the present application; Figure 6 As shown, optionally, in step S102, determining scene depth information corresponding to each scene background image according to each scene background image collected may include: S401 . Encode a scene background image using a depth estimation model and a segmentation model to obtain a feature map of the scene background image.

[0082] In some embodiments, for a scene background image, the scene background image may be used as input, and encoded by a depth estimation model Depth-Anything and a segmentation model DINOV2 to obtain a feature map.

[0083] S402: Decode the feature map of the scene background image to obtain a depth map of the scene background image.

[0084] Then input Depth-Anything to decode the feature map obtained above to obtain the depth map of the scene background image. Its purpose is to use the discrete information of semantic segmentation to guide the depth estimation model to learn the depth representation of different objects, thereby enhancing the accuracy of monocular depth estimation.

[0085] S403: Determine depth information of the scene background image according to the depth map.

[0086] Since each pixel in the depth map contains depth information, the depth map is an image representation of depth information. Therefore, the depth information of the scene background image can be determined based on the depth map. The depth information of the scene background image contains the depth information of each pixel in the scene background image. This depth information indicates the distance between the pixel and the camera emission source.

[0087] Figure 7 A flowchart of another image processing method provided in an embodiment of the present application; Figure 7 As shown, optionally, in step S103, determining the target area in each scene background image according to each scene background image may include: S501: Generate a road surface mask image in the scene background image according to the scene background image.

[0088] In some embodiments, a Grounding-DINO (Advanced Open Detection Model) model may be used to process the scene background image to generate a road surface mask image in the scene background image.

[0089] Alternatively, the road surface mask image may be extracted from the scene background image based on the large language vision model by referring to the above-mentioned method for extracting the foreground object image.

[0090] S502: Determine an embeddable area in the scene background image according to the road mask image.

[0091] According to the road surface mask image, the embeddable area in the scene background image can be calculated and generated. The embeddable area can indicate the maximum inscribed bounding box of the road surface mask image. The embeddable area can be used as the effective area of ​​the road surface. Within this area, the foreground object can be embedded, and the remaining area outside the embeddable area in the scene background image can be excluded. The purpose of determining the embeddable area is to exclude non-valid areas such as the edge of the road surface.

[0092] S503: Determine area ratio information according to the boundary information of the embeddable area and the width information of the scene background image.

[0093] The area ratio information is used to represent the ratio of the target area to be selected to the embeddable area.

[0094] Based on the determined boundary information of the embeddable area and the width information of the scene background image, the area ratio information (also called ROI) can be calculated. ratio ).

[0095] The boundary information here may refer to information of a boundary box constituting the embeddable area, and the width information of the scene background image may refer to the side length of the scene background image in the vertical direction.

[0096] S504: Determine the region segmentation position according to the region ratio information and the boundary information of the embeddable region, and segment the embeddable region based on the region segmentation position to obtain the target region.

[0097] Next, the region segmentation position can be determined based on the calculated region ratio information combined with the boundary information of the embeddable region. Among them, the coordinates of the segmentation point when performing region segmentation can be determined first, so as to make a horizontal segmentation line based on the segmentation point, and the embeddable region can be divided into two parts based on the horizontal segmentation line. Since there is no need to detect objects that are too far away from the camera, the foreground object does not need to be embedded in a farther area when embedded, so the area closer to the camera in the two parts obtained by segmentation can be used as the target area, that is, the part closer to the bottom edge of the scene background image can be used as the target area.

[0098] Figure 8 A flowchart of another image processing method provided in an embodiment of the present application; Fig. 9 A schematic diagram of a road surface mask of a scene background image provided in an embodiment of the present application, Fig.10 A schematic diagram of an embeddable area of ​​a scene background image provided in an embodiment of the present application. Fig.10 The first boundary line, the second boundary line, the third boundary line and the fourth boundary line are respectively identified. Figure 8-Figure 10 As shown, optionally, in step S502, determining the embeddable area in the scene background image according to the road surface mask image may include: S601: Determine, according to a road surface mask image, a first distance between a road surface and a bottom edge of a scene background image and a second distance between a road surface and a top edge of the scene background image.

[0099] According to the road mask image, the first distance of the road closest to the bottom edge of the scene background image and the second distance of the road closest to the top edge of the scene background image can be determined on the basis of removing non-road information in the mask image.

[0100] by Fig. 9 Taking the road mask image shown in FIG. 1 as an example, since there is text information in the lower right corner of the mask image, the text information needs to be removed from the mask image. After removing the text information, the determined first distance can be as follows: Fig.10 Of course, if there is no non-road surface information in the lower right corner of the road surface mask image, the vertical coordinate of the bottom edge boundary line of the road surface mask image can be directly determined as the first distance.

[0101] The second distance is determined according to the farthest boundary line of the road mask image. The second distance can be as follows: Fig.10 The distance y2 marked in . The coordinate system of the image takes the lower left corner of the image as the origin, the length of the image is the x-axis, and the width of the image is the y-axis.

[0102] S602: Determine a first boundary line according to the first distance and the boundary of the road surface mask image.

[0103] According to the first distance, parallel lines are made parallel to the upper and lower sides of the scene background image, and the length of the parallel lines needs to be inscribed in the bottom boundary of the road mask image, that is, a horizontal line inscribed in the bottom boundary line of the road mask image is made to obtain the first boundary line.

[0104] S603: Determine a second boundary line according to the second distance and the boundary of the road surface mask image.

[0105] Similarly, based on the second distance, a horizontal line is drawn to inscribe the top edge boundary line of the road surface mask image to obtain a second boundary line.

[0106] S604: Determine a third boundary line and a fourth boundary line respectively according to the first boundary line, the second boundary line, and the image area of ​​the road surface mask image.

[0107] A ray is emitted from the left end point of the second boundary line to the first boundary line, and the maximum ray that can cover the left boundary line of the road surface mask image is determined as the third boundary line. Similarly, a ray is emitted from the right end point of the second boundary line to the first boundary line, and the maximum ray that can cover the right boundary line of the road surface mask image is determined as the fourth boundary line.

[0108] S605: The first boundary line, the second boundary line, the third boundary line and the fourth boundary line constitute an embeddable area.

[0109] The first boundary line, the second boundary line, the third boundary line and the endpoints of the third boundary line are connected in sequence to form a closed figure, and then an embeddable area can be obtained.

[0110] Fig.10 The embeddable regions in Fig. 9 The embeddable area is determined by the road surface mask image in the image. When the road surface mask image changes, the determined embeddable area will also adapt to the change, but the embeddable area can be generated in the above manner.

[0111] Optionally, in step S503, determining the area ratio information based on the boundary information of the embeddable area and the width information of the scene background image may include: determining the area ratio information based on the endpoint coordinate information of the first boundary line of the embeddable area, the endpoint coordinate information of the second boundary line and the width information of the scene background image.

[0112] Optionally, the area ratio information may be calculated using a preset calculation formula according to the horizontal coordinate of the left endpoint of the first boundary line, the horizontal coordinate of the left endpoint of the second boundary line, and width information of the scene background image.

[0113] Continue to refer Fig.10 , assuming that the horizontal coordinate of the left endpoint of the first boundary line is , the horizontal coordinate of the left endpoint of the second boundary line is , the width information of the scene background image is , the regional ratio information ROI can be calculated using the following formula ratio :

[0114] Optionally, in step S504, the region segmentation position is determined according to the region proportion information and the boundary information of the embeddable region, and the embeddable region is segmented based on the region segmentation position. Before obtaining the target area, it also includes: if the region proportion information is greater than a preset threshold, the region proportion information is adjusted to obtain the adjusted region proportion information.

[0115] In some embodiments, when the endpoint of the second boundary line of the generated embeddable area is close to the left edge or the right edge, Fig.10 The embeddable area shown in will change from a trapezoid to a rectangle. This will result in the ROI calculated using the above formula ratio The value becomes larger, so the final calculated target area becomes smaller.

[0116] To solve this problem, the calculated value can be adjusted using the following formula:

[0117] That is, if the calculated ROI is ratio When it exceeds 0.5, the operation (1-ROI ratio ) to effectively compensate for the ROI ratio The problem is that the target area gradually shrinks as the

[0118] Optionally, in step S504, determining the region segmentation position according to the region ratio information and the boundary information of the embeddable region may include: determining the region segmentation position according to the endpoint coordinate information of the first boundary line of the embeddable region, the endpoint coordinate information of the second boundary line, and the region ratio information.

[0119] Next, the ROI can be calculated based on the ordinate of the left end point of the first boundary line of the embeddable area and the ordinate of the left end point of the second boundary line, combined with the adjusted area ratio information ROI. ratio, calculate the vertical coordinate of the region segmentation position point.

[0120] Assume that the ordinate of the left endpoint of the first boundary line is , the ordinate of the left endpoint of the second boundary line is , the vertical coordinate of the region segmentation point can be calculated using the following formula:

[0121] Based on the vertical coordinates of the region segmentation points, horizontal lines can be drawn to generate region segmentation lines. Fig.10 , assuming that the region segmentation line is ROI top , then, by ROI top The area formed by the first boundary line, the third boundary line and the third boundary line of the embeddable area is the target area. Fig.10 The shaded area in .

[0122] Fig.11 A flowchart of another image processing method provided in an embodiment of the present application; Fig.11 As shown, optionally, in step S104, embedding at least one foreground object image into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate multiple composite images may include: S701, determining a target embedding position in a target area in a target scene background image; the target embedding position is any position in the target area.

[0123] It is worth noting that any position in the determined target area can be used as the target embedding position, that is, the foreground object image can be embedded in any position in the target area.

[0124] S702: Determine depth information of a target embedding position according to scene depth information corresponding to a target scene background image.

[0125] Based on the determined target embedding position, depth information of the target embedding position may be determined according to scene depth information.

[0126] S703: Determine the image size of the target foreground object at the target embedding position in the target scene background image to be embedded according to the depth information of the target embedding position and the mapping relationship between the depth information and the image embedding size.

[0127] Optionally, based on the display rule that objects are larger when they are closer to the camera and smaller when they are farther away from the camera, that is, the closer the object is to the camera, the larger the display size of the object is, and the farther the object is from the camera, the smaller the display size of the object is, the mapping relationship between the depth information and the image embedding size can be pre-built.

[0128] Then, based on the depth information of the target embedding position and the mapping relationship between the depth information and the image embedding size, the size that the foreground object image should have when embedded in the target embedding position in the scene background image can be determined.

[0129] S704: embed the target foreground object image into the target embedding position in the target scene background image according to the image size to generate a composite image.

[0130] Based on the determined image size, the target foreground object image can be embedded into the target embedding position with the image size to generate a composite image.

[0131] The target foreground object image may be any extracted foreground object image, and the target scene background image may be any scene background image.

[0132] In some embodiments, in order to increase the number of generated composite images, each foreground object image may be rotated according to a preset rotation angle to expand and obtain more foreground object images.

[0133] Therefore, the expanded foreground object image can also be embedded into the scene background image to generate a composite image.

[0134] Fig.12 A flowchart of another image processing method provided by an embodiment of the present application; Optionally, in step S104, embedding at least one foreground object image into at least one scene background image to generate multiple composite images may include: S801. Embed one or more of the foreground object images into a current scene background image to generate at least one synthetic image corresponding to the current scene background image, where the current scene background image is any scene background image.

[0135] In one feasible manner, for a scene background image, one or more of all the extracted foreground object images may be used as embedded images and embedded into the scene background image to obtain at least one composite image corresponding to the scene background image.

[0136] That is, a foreground object image can be embedded in one scene background image or in different scene background images, and a scene background image can be embedded with only one foreground object image or with multiple identical or different foreground object images, as long as the final synthesized images are not completely identical.

[0137] S802: Obtain multiple composite images according to at least one composite image corresponding to each scene background image.

[0138] Each scene background image corresponds to at least one synthetic image, thereby forming a plurality of synthetic images.

[0139] Optionally, in step S104, at least one foreground object image is embedded in at least one scene background image to generate multiple composite images, which may include: if the number of foreground object images currently embedded in the target scene background image does not exceed a preset upper limit on the number of embeddings, then according to the embedding positions of the currently embedded foreground object images, the foreground object image to be embedded is embedded in the target scene background image to generate a current composite image.

[0140] If multiple foreground object images need to be embedded in a scene background image, in order to avoid overlap and occlusion between the embedded foreground object images, an upper limit on the number of foreground object images that can be embedded in the target area can be set. Each time the current foreground object image is embedded, the number of foreground object images already embedded in the scene background image can be first determined. If the upper limit is not reached, the current foreground object image can be embedded, and the current foreground object image needs to be embedded in a new non-overlapping position according to the embedding positions of the embedded foreground object images. If the upper limit is reached, the current foreground object image cannot be embedded.

[0141] Optionally, in step S104, after generating a plurality of composite images, the method further includes: using a pre-trained image harmonization model to repair inharmonious regions in each composite image to obtain an optimized composite image.

[0142] In some embodiments, the composite image obtained by the above method may have defects, such as the foreground and background look disharmonious, the interlacing is unnatural, etc. To solve this problem, a pre-trained image harmonization model can be used to repair the composite image to adjust the foreground object or background in the composite image to make the composite image more harmonious.

[0143] Optionally, the training process of the image harmonization model may include: collecting a training sample data set, the training sample data set includes: multiple groups of sample data, each group of sample data includes: real captured images and synthetic images constructed based on the real captured images; using the training sample data set to train the image harmonization model.

[0144] In some embodiments, multiple real images can be collected from a database, and based on the real images, synthetic images corresponding to the real images can be constructed, so that one real image and one synthetic image can constitute a set of sample data, and an image harmonization model can be trained from multiple sets of sample data.

[0145] In the process of training the image harmonization model, we tried the concept of domain verification and used DoveNet (Domain VErification Network, a deep learning model for image harmonization) based on domain verification. Specifically, we believe that the foreground object and background of the synthetic image are collected in different shooting environments, and each shooting environment can be regarded as a domain. In reality, there may be countless domains, and we do not know the domain labels of the foreground object and the background, so we can only migrate the foreground object to the same domain as the background.

[0146] In summary, the image processing method provided by this embodiment includes: extracting at least one foreground object image from each acquired initial image; determining the scene depth information corresponding to each scene background image according to each acquired scene background image; determining the target area in each scene background image according to each scene background image; embedding at least one foreground object image into at least one scene background image according to the scene depth information corresponding to each scene background image and the target area in each scene background image to generate multiple synthetic images. Based on the foreground object image extracted from the initial image, combined with the depth information of the scene background image and the target area of ​​the scene background image, the method can determine the image size and embedding position of the foreground object image when embedded in the scene background image, so as to embed the foreground object image into the scene background image according to a reasonable size and position, and obtain a synthetic image in a traffic scene. It realizes replacing expensive manual annotation data with the method of generating synthetic images. Since the foreground object in the synthetic image has a clear identification, it can provide accurate label information for the training of the downstream detection model. On the premise of reducing the time and labor of manual annotation, the training accuracy of the downstream detection model can also be improved.

[0147] In addition, extracting the foreground object image from the initial image based on the description information can improve the accuracy of the foreground object image extraction result and facilitate the generation of annotation information of the foreground object image.

[0148] The following describes the apparatus, device, storage medium, etc. used to execute the image processing method provided by the present application. The specific implementation process and technical effects are described above and will not be repeated below.

[0149] Fig.13 The following is a schematic diagram of an image processing device provided in an embodiment of the present application. The functions implemented by the image processing device correspond to the steps performed by the above method. The device can be understood as the above terminal, server, or server processor, or can also be understood as a component independent of the above server or processor that implements the functions of the present application under the control of the server, such as Fig.13 As shown, the device includes: an acquisition module 130, a determination module 131, and an embedding module 132; An acquisition module 130, configured to extract at least one foreground object image from each acquired initial image; A determination module 131 is used to determine scene depth information corresponding to each scene background image according to the collected scene background images; A determination module 131, configured to determine a target area in each scene background image according to each scene background image; The embedding module 132 is used to embed at least one foreground object image into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate a plurality of composite images.

[0150] Optionally, the acquisition module 130 is specifically configured to collect a plurality of initial images, each of which contains at least one foreground object; According to each initial image and at least one foreground object description information corresponding to each initial image, a large language visual model is used to extract each foreground object image corresponding to each foreground object description information from each initial image.

[0151] Optionally, the acquisition module 130 is specifically configured to encode the initial image and the target foreground object description information corresponding to the initial image by the large language visual model to generate fused encoding information; Segment and decode the fused coding information, extract the binary mask image of the target foreground object corresponding to the target foreground object description information from the initial image, and transform the binary mask image to obtain the bounding box of the target foreground object; According to the bounding box of the target foreground object, the target foreground object image is extracted.

[0152] Optionally, the determination module 131 is specifically configured to respectively use a depth estimation model and a segmentation model to encode the scene background image to obtain a feature map of the scene background image; Decoding the feature map of the scene background image to obtain a depth map of the scene background image; Determine the depth information of the scene background image based on the depth map.

[0153] Optionally, the determination module 131 is specifically configured to generate a road surface mask image in the scene background image according to the scene background image; Determine an embeddable area in the scene background image according to the road mask image; Determine the region ratio information according to the boundary information of the embeddable region and the width information of the scene background image; the region ratio information is used to represent the ratio of the target region to be selected to the embeddable region; The region segmentation position is determined according to the region ratio information and the boundary information of the embeddable region, and the embeddable region is segmented based on the region segmentation position to obtain the target region.

[0154] Optionally, the determination module 131 is specifically configured to determine, according to the road surface mask image, a first distance between the road surface and the bottom edge of the scene background image and a second distance between the road surface and the top edge of the scene background image; Determining a first boundary line according to the first distance and a boundary of the road surface mask image; Determining a second boundary line according to the second distance and the boundary of the road surface mask image; Determine a third boundary line and a fourth boundary line respectively according to the first boundary line, the second boundary line and the image area of ​​the road surface mask image; The first boundary line, the second boundary line, the third boundary line and the fourth boundary line constitute an embeddable area.

[0155] Optionally, the determination module 131 is specifically configured to determine the area ratio information according to the endpoint coordinate information of the first boundary line of the embeddable area, the endpoint coordinate information of the second boundary line, and the width information of the scene background image.

[0156] Optionally, an adjustment module is also included; The adjustment module is used to adjust the area ratio information if the area ratio information is greater than a preset threshold to obtain the adjusted area ratio information.

[0157] Optionally, the determination module 131 is specifically configured to determine the region segmentation position according to the endpoint coordinate information of the first boundary line of the embeddable region, the endpoint coordinate information of the second boundary line, and the region ratio information.

[0158] Optionally, the embedding module 132 is specifically used to determine the target embedding position from the target area in the target scene background image; the target embedding position is any position in the target area; Determine the depth information of the target embedding position according to the scene depth information corresponding to the target scene background image; Determine the image size of the target foreground object at the target embedding position in the target scene background image to be embedded according to the depth information of the target embedding position and the mapping relationship between the depth information and the image embedding size; According to the image size, the target foreground object image is embedded at the target embedding position in the target scene background image to generate a composite image.

[0159] Optionally, an embedding module 132 is specifically configured to embed one or more of the foreground object images into a current scene background image to generate at least one synthetic image corresponding to the current scene background image, where the current scene background image is any one of the scene background images; A plurality of composite images are obtained according to at least one composite image corresponding to each scene background image.

[0160] Optionally, the embedding module 132 is specifically used to embed the foreground object image to be embedded into the target scene background image according to the embedding position of each currently embedded foreground object image if the number of foreground object images currently embedded in the target scene background image does not exceed a preset embedding number upper limit, so as to generate a current composite image.

[0161] Optionally, the adjustment module is further used to use a pre-trained image harmonization model to repair the inharmonious areas in each composite image to obtain an optimized composite image.

[0162] Optionally, it also includes: a training module; A training module is used to collect a training sample data set, the training sample data set includes: multiple groups of sample data, each group of sample data includes: a real shot image and a synthetic image constructed based on the real shot image; The image harmonization model is trained using a training sample data set.

[0163] The above-mentioned device is used to execute the method provided by the aforementioned embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0164] The above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more digital singnal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a module is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0165] The above modules can be connected or communicated with each other via a wired connection or a wireless connection. The wired connection may include a metal cable, an optical cable, a hybrid cable, etc., or any combination thereof. The wireless connection may include a connection in the form of a LAN, a WAN, Bluetooth, a ZigBee, or NFC, or any combination thereof. Two or more modules may be combined into a single module, and any one module may be divided into two or more units. Those skilled in the art will clearly understand that for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application.

[0166] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, which may be a computing device with data processing capabilities.

[0167] The device may include: a processor 801 and a storage medium 802 .

[0168] The storage medium 802 is used to store programs, and the processor 801 calls the programs stored in the storage medium 802 to execute the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0169] The storage medium 802 stores program code, and when the program code is executed by the processor 801, the processor 801 executes various steps of the image processing method according to various exemplary embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0170] The processor 801 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and may implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0171] The storage medium 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The storage medium may include at least one type of storage medium, for example, a flash memory, a hard disk, a multimedia card, a card-type storage medium, a random access storage medium (Random Access Memory, RAM), a static random access storage medium (Static Random Access Memory, SRAM), a programmable read-only storage medium (Programmable Read Only Memory, PROM), a read-only storage medium (Read Only Memory, ROM), an electrically erasable programmable read-only storage medium (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic storage medium, a disk, an optical disk, and the like. The storage medium is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The storage medium 802 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0172] Optionally, the present application also provides a program product, such as a computer-readable storage medium, including a program, which is used to execute the above method embodiment when executed by a processor.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0176] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only storage medium (English: Read-Only Memory, referred to as: ROM), random access storage medium (English: Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

Claims

1. An image processing method, characterized in that: include: extracting at least one foreground object image from each acquired initial image; Determine the scene depth information corresponding to each scene background image according to the collected background images of each scene; Determine the target area in each scene background image according to each scene background image; At least one foreground object image is embedded into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate a plurality of composite images.

2. The method according to claim 1, characterized in that The step of extracting at least one foreground object image from each of the acquired initial images comprises: Collecting a plurality of initial images, each of which contains at least one foreground object; According to each initial image and at least one foreground object description information corresponding to each initial image, a large language visual model is used to extract each foreground object image corresponding to each foreground object description information from each initial image.

3. The method according to claim 2, characterized in that The extracting, based on each initial image and at least one foreground object description information corresponding to each initial image, each foreground object image corresponding to each foreground object description information from each initial image using a large language visual model comprises: The large language visual model encodes the initial image and the target foreground object description information corresponding to the initial image to generate fused encoding information; Segmenting and decoding the fused coding information, extracting a binary mask image of the target foreground object corresponding to the target foreground object description information from the initial image, and converting the binary mask image to obtain a bounding box of the target foreground object; According to the bounding box of the target foreground object, a target foreground object image is extracted.

4. The method according to claim 1, characterized in that: The step of determining scene depth information corresponding to each scene background image according to the collected scene background images includes: A depth estimation model and a segmentation model are respectively used to encode the scene background image to obtain a feature map of the scene background image; Decoding the feature map of the scene background image to obtain a depth map of the scene background image; Determine depth information of the scene background image according to the depth map.

5. The method according to claim 1, characterized in that The step of determining the target area in each scene background image according to each scene background image includes: Generate a road surface mask image in the scene background image according to the scene background image; Determining an embeddable area in the scene background image according to the road mask image; Determine region ratio information according to the boundary information of the embeddable region and the width information of the scene background image; the region ratio information is used to characterize the ratio of the target region to be selected to the embeddable region; The region segmentation position is determined according to the region ratio information and the boundary information of the embeddable region, and the embeddable region is segmented based on the region segmentation position to obtain the target region.

6. The method according to claim 5, characterized in that The step of determining the embeddable area in the scene background image according to the road mask image includes: Determining, according to the road mask image, a first distance of the road surface close to the bottom edge of the scene background image and a second distance of the road surface close to the top edge of the scene background image; Determining a first boundary line according to the first distance and a boundary of the road surface mask image; determining a second boundary line according to the second distance and a boundary of the road surface mask image; Determine a third boundary line and a fourth boundary line respectively according to the first boundary line, the second boundary line and the image area of ​​the road surface mask image; The embeddable area is formed by the first boundary line, the second boundary line, the third boundary line, and the fourth boundary line.

7. The method according to claim 6, characterized in that The determining of area ratio information according to the boundary information of the embeddable area and the width information of the scene background image includes: The area ratio information is determined according to the endpoint coordinate information of the first boundary line of the embeddable area, the endpoint coordinate information of the second boundary line, and the width information of the scene background image.

8. The method according to claim 5, characterized in that Before determining the region segmentation position according to the region ratio information and the boundary information of the embeddable region, and segmenting the embeddable region based on the region segmentation position to obtain the target region, the method further includes: If the area ratio information is greater than a preset threshold, the area ratio information is adjusted to obtain adjusted area ratio information.

9. The method according to claim 6, characterized in that The determining of the region segmentation position according to the region ratio information and the boundary information of the embeddable region includes: The area division position is determined according to the endpoint coordinate information of the first boundary line of the embeddable area, the endpoint coordinate information of the second boundary line, and the area ratio information.

10. The method according to claim 1, characterized in that The method embedding at least one foreground object image into at least one scene background image according to scene depth information corresponding to each scene background image and a target area in each scene background image to generate a plurality of composite images includes: Determine a target embedding position from a target area in a target scene background image; the target embedding position is any position in the target area; Determining the depth information of the target embedding position according to the scene depth information corresponding to the target scene background image; Determine the image size of the target foreground object to be embedded in the target scene background image at the target embedding position according to the depth information of the target embedding position and the mapping relationship between the depth information and the image embedding size; According to the image size, the target foreground object image is embedded in the target embedding position in the target scene background image to generate a composite image.

11. The method according to claim 1, characterized in that: The step of embedding at least one foreground object image into at least one scene background image to generate a plurality of composite images comprises: embedding one or more of the foreground object images into a current scene background image to generate at least one synthetic image corresponding to the current scene background image, wherein the current scene background image is any scene background image; The multiple composite images are obtained according to at least one composite image corresponding to each scene background image.

12. The method according to claim 1, characterized in that The step of embedding at least one foreground object image into at least one scene background image to generate a plurality of composite images comprises: If the number of foreground object images currently embedded in the target scene background image does not exceed the preset upper limit of the embedding number, the foreground object image to be embedded is embedded into the target scene background image according to the embedding position of each currently embedded foreground object image to generate a current composite image.

13. The method according to any one of claims 1 to 12, characterized in that: After generating a plurality of composite images, the method further includes: The pre-trained image harmonization model is used to repair the inharmonious areas in each composite image to obtain an optimized composite image.

14. The method according to claim 13, characterized in that The training process of the image harmonization model includes: Collecting a training sample data set, the training sample data set comprising: a plurality of groups of sample data, each group of sample data comprising: a real shot image and a synthetic image constructed based on the real shot image; The image harmonization model is obtained by training using the training sample data set.

15. An image processing device, characterized in that: include: Get modules, determine modules, embed modules; The acquisition module is used to extract at least one foreground object image from each acquired initial image; The determination module is used to determine the scene depth information corresponding to each scene background image according to the collected scene background images; The determination module is used to determine the target area in each scene background image according to each scene background image; The embedding module is used to embed at least one foreground object image into at least one scene background image according to the scene depth information corresponding to each scene background image and the target area in each scene background image to generate multiple composite images.

16. An electronic device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor and the storage medium communicate via the bus, and the processor executes the program instructions to implement the image processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the image processing method according to any one of claims 1 to 14 is implemented.

Images