Image processing method and device, equipment, medium and program product

Abstract

The embodiment of the invention provides an image processing method and device, equipment, a medium and a program product. The image processing method comprises the following steps: determining a to-be-processed original image; according to the depth value of the original image, a background part in the original image is removed, a target image is obtained, and the number of object detection frames in the target image is larger than or equal to the predetermined number of main objects in the original image; semantic segmentation is carried out on the target image, a plurality of object images are obtained, and each object image comprises an object; and according to the feature information corresponding to each object image, the object images are screened, the number of the screened object images is greater than or equal to the number of the main objects, and object identification is carried out based on the screened object images. By applying the technical scheme provided by the embodiment of the invention, the probability of correctly identifying the object in the image can be improved, and the identification efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to an image processing method, device, equipment, medium and program product. Background Art

[0002] In the current field of storyboards, there are certain challenges in controlling the consistency of characters. It is difficult to ensure that the facial features and hairstyles of the same character have high similarity in multiple images generated continuously. For the same character with low similarity in different images, it is necessary to redraw it. Especially in the storyboard synthesis business, it is crucial to correctly redraw the characters in the image.

[0003] At present, the multimodal large model can analyze the image and the prompt information corresponding to the image to identify the person in the image, determine the correspondence between the person in the image and the person information in the prompt information, and then redraw the person in the image based on these correspondences. However, if the image expresses complex scene content, it will be difficult for the multimodal large model to correctly identify the person in the image.

[0004] Therefore, how to improve the probability of correctly identifying objects in images is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method, device, equipment, medium and program product, which can improve the probability of correctly identifying objects in images. The specific technical solution is as follows:

[0006] In a first aspect, an image processing method is provided, comprising:

[0007] Determine the original image to be processed;

[0008] According to the depth value of the original image, a background portion of the original image is removed to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to a predetermined number of main objects in the original image;

[0009] Performing semantic segmentation on the target image to obtain a plurality of object images, each of which contains an object;

[0010] The object images are screened according to the feature information corresponding to each object image, and the number of the screened object images is greater than or equal to the number of the main objects.

[0011] Optionally, removing a background portion of the original image according to the depth value of the original image to obtain a target image includes:

[0012] Performing depth estimation on the original image to determine a depth value of the original image;

[0013] determining an initial depth threshold as a target depth threshold;

[0014] Based on the target depth threshold, the depth value of the original image is filtered to remove the background portion of the original image to obtain a process image;

[0015] Performing target detection on the process image to obtain an object detection frame in the process image;

[0016] In a case where the number of object detection frames in the process image is greater than or equal to the number of main objects, the process image is determined as a target image.

[0017] Optionally, when the number of object detection boxes in the process image is less than the number of main objects, the method further comprises:

[0018] Adjusting the target depth threshold;

[0019] Repeat the step of filtering the depth value of the original image based on the target depth threshold, removing the background portion of the original image, and obtaining a process image.

[0020] Optionally, the adjusting the target depth threshold comprises:

[0021] If the depth value corresponding to the background part in the original image is less than the depth value corresponding to the foreground part, reducing the target depth threshold according to a preset first step length;

[0022] If the depth value corresponding to the background portion in the original image is greater than the depth value corresponding to the foreground portion, the target depth threshold is increased according to a preset second step size.

[0023] Optionally, removing a background portion of the original image according to the depth value of the original image to obtain a target image includes:

[0024] Performing target detection on the original image to obtain an object detection frame in the original image;

[0025] When the number of object detection frames in the original image is greater than or equal to the number of main objects, the background portion of the original image is removed according to the depth value of the original image to obtain a target image.

[0026] Optionally, screening the object images according to the feature information corresponding to each object image includes:

[0027] Determine feature information corresponding to each object image, wherein the feature information corresponding to each object image includes image area and object posture;

[0028] Let i = 1, and filter the object images corresponding to the i-th object posture in the order of image area from large to small;

[0029] If the number of the screened object images is greater than or equal to the number of the main objects, stop screening;

[0030] If the number of the screened object images is less than the number of the main objects, let i=i+1, and repeat the step of screening the object images corresponding to the i-th object posture in the order of image area from large to small;

[0031] Among them, the priority of the object image corresponding to the i-th object posture is higher than the priority of the object image corresponding to the (i+1)-th object posture.

[0032] Optionally, after filtering the object images according to the feature information corresponding to each object image, the method further includes:

[0033] Adjusting other parts of the filtered object image except the object to a preset color;

[0034] The screened object images are merged into one image according to a preset picture sequence, and there is a preset interval between each object image in the merged image.

[0035] Optionally, the number of main objects in the original image is determined by the following steps:

[0036] Obtaining prompt information of the original image;

[0037] According to the prompt information, the number of main objects in the original image is determined.

[0038] In a second aspect, an image processing device is provided, comprising:

[0039] A determination module, used for determining an original image to be processed;

[0040] A first obtaining module is used to remove the background part of the original image according to the depth value of the original image to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to the predetermined number of main objects in the original image;

[0041] A second acquisition module is used to perform semantic segmentation on the target image to obtain a plurality of object images, each of which contains an object;

[0042] The screening module is used to screen the object images according to the feature information corresponding to each object image, and the number of the screened object images is greater than or equal to the number of the main objects.

[0043] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0044] Memory, used to store computer programs;

[0045] The processor is used to implement the steps of the image processing method described in the first aspect when executing the program stored in the memory.

[0046] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, the steps of the image processing method described in the first aspect are implemented.

[0047] In a fifth aspect, a computer program product is provided, which includes computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that an electronic device having the processor performs the steps of the image processing method described in the first aspect.

[0048] By applying the technical solution provided in the embodiment of the present application, after obtaining the original image to be processed, the background part of the original image is removed according to the depth value of the original image to obtain the target image, and the target image is semantically segmented to obtain multiple object images, each of which contains an object, and the object images are screened according to the feature information corresponding to each object image. That is, the original image is first processed to screen out the object images, and each screened object image contains an object, so that object recognition can be performed based on the screened object images. At the same time, the number of screened object images is greater than or equal to the number of main objects, so that the recognized objects can be better matched with the main objects in the original image. In addition, the screened object images do not contain the background part of the original image, which reduces the interference of the background part in the scene content expressed by the original image on the main object, so that when performing object recognition, more focus can be placed on analyzing and understanding the main objects in the original image, which can increase the probability of correct recognition of objects in the image and improve recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0050] Figure 1 This is a flowchart of an implementation of an image processing method in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of a depth map in an embodiment of the present application;

[0052] Figure 3 is a schematic diagram of an object detection frame in an image in an embodiment of the present application;

[0053] Figure 4 is a schematic diagram of the merged image in the embodiment of the present application;

[0054] Figure 5 This is a schematic diagram of the structure of an image processing device in an embodiment of the present application;

[0055] Figure 6 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0057] The core of this application is to provide an image processing method, which can be applied to scenarios such as image processing, film and television production, and storyboard synthesis business.

[0058] See also Figure 1 FIG. 1 is a flowchart of an implementation of an image processing method provided in an embodiment of the present application. The method may include the following steps:

[0059] S110: Determine the original image to be processed.

[0060] In the embodiment of the present application, the original image may be any image to be processed. Optionally, the original image may be an image generated by an artificial intelligence (AI) model, for example, in film and television production, a film and television image generated by an AI model is used as the original image.

[0061] When there is a need for image processing, the original image to be processed can be determined first. Optionally, the original image to be processed can be determined according to the processing instructions of the user, or, after the AI ​​model generates a film and television image, the film and television image generated by the AI ​​model is automatically determined as the original image to be processed.

[0062] S120: According to the depth value of the original image, the background portion of the original image is removed to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to the predetermined number of main objects in the original image.

[0063] The original image expresses the content of a certain scene, and may include a variety of image elements such as people, animals, and buildings in the original image. The objects in the embodiments of the present application may include people or animals. The main objects in the original image can be understood as the main objects that need to be identified among the objects contained in the original image. For example, if the original image is a film or television image, the main objects in the original image may include the protagonist and important supporting roles.

[0064] The number of main objects in the original image may be determined in advance, for example, based on image information input by a user, or based on relevant information of the original image.

[0065] After determining the original image to be processed, the depth value of the original image can be determined. Each pixel of the original image can correspond to a depth value, such as a value between 0 and 255. The depth value of the original image can be understood as a set of depth values ​​corresponding to each pixel.

[0066] According to the depth value of the original image, the background part of the original image can be determined. The depth values ​​corresponding to the background part and the foreground part may be different due to different image encoding methods. For example, in one image encoding method, the depth value corresponding to the background part is greater than the depth value corresponding to the foreground part, that is, the larger the depth value, the greater the possibility that it corresponds to the background part. For another example, in another image encoding method, the depth value corresponding to the background part is less than the depth value corresponding to the foreground part, that is, the smaller the depth value, the greater the possibility that it corresponds to the background part.

[0067] After determining the background part in the original image, the background part in the original image can be removed to obtain the target image. The number of object detection boxes in the target image must be greater than or equal to the number of main objects in the original image, which helps to ensure the accuracy and completeness of the object recognition in the subsequent steps.

[0068] S130: Perform semantic segmentation on the target image to obtain multiple object images, each of which contains an object.

[0069] In an embodiment of the present application, the original image to be processed is determined, and the background part of the original image is removed according to the depth value of the original image. After obtaining the target image, the target image can be further semantically segmented. Optionally, a semantic segmentation algorithm, such as a yolov8 model, can be used to semantically segment the target image to obtain multiple object images. Each object image contains an object.

[0070] S140: Filtering the object images according to the feature information corresponding to each object image, wherein the number of the filtered object images is greater than or equal to the number of main objects, so as to perform object recognition based on the filtered object images.

[0071] In an embodiment of the present application, after obtaining multiple object images, feature information corresponding to each object image can be determined, and based on the feature information corresponding to each object image, the object images can be screened to select object images that meet processing requirements.

[0072] The number of the screened object images must be greater than or equal to the number of the main objects, so that the screened object images can include the main objects in the original image, which helps to improve the accuracy of object recognition.

[0073] After the object images are screened out, object recognition can be performed based on the screened object images. Optionally, a pre-trained large model can be used to perform object recognition based on the screened object images. Further, after the large model recognizes the main object, the correspondence between the recognized main object and the object information in the prompt information of the original image can be determined, and the main object in the original image can be redrawn based on these correspondences.

[0074] By applying the method provided in the embodiment of the present application, after obtaining the original image to be processed, the background part of the original image is removed according to the depth value of the original image to obtain the target image, and the target image is semantically segmented to obtain multiple object images, each of which contains an object, and the object images are screened according to the feature information corresponding to each object image. That is, the original image is first processed to screen out the object images, and each screened object image contains an object, so that object recognition can be performed based on the screened object images. At the same time, the number of screened object images is greater than or equal to the number of main objects, so that the recognized objects can be better matched with the main objects in the original image. In addition, the screened object images do not contain the background part of the original image, which reduces the interference of the background part in the scene content expressed by the original image on the main object, so that when performing object recognition, more attention can be paid to analyzing and understanding the main objects in the original image, which can increase the probability of correct recognition of objects in the image and improve recognition efficiency.

[0075] In some embodiments of the present application, step S120 removes the background portion of the original image according to the depth value of the original image to obtain the target image, which may include the following steps:

[0076] Performing depth estimation on the original image to determine the depth value of the original image;

[0077] determining an initial depth threshold as a target depth threshold;

[0078] Compare each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image;

[0079] Filter out the depth value corresponding to the background part of the original image to remove the background part in the original image and obtain a process image;

[0080] Performing target detection on the process image to obtain the object detection frame in the process image;

[0081] In the case where the number of object detection frames in the process image is greater than or equal to the number of main objects, the process image is determined as a target image.

[0082] For the convenience of description, the above steps are combined for explanation.

[0083] In the embodiment of the present application, an initial depth threshold may be preset, and the initial depth threshold may be set based on experience or historical data.

[0084] After determining the original image to be processed, the depth of the original image can be estimated. Optionally, a depth estimation algorithm, such as depth-anything, can be used to estimate the depth of the original image. By performing depth estimation on the original image, the depth value of the original image can be obtained. For example, the depth map corresponding to the original image is as follows: Figure 2 shown.

[0085] The initial depth threshold is determined as the target depth threshold, and then each depth value of the original image is compared with the target depth threshold to determine the depth value corresponding to the background part of the original image, and the depth value corresponding to the background part of the original image is filtered out to remove the background part in the original image to obtain a process image. Optionally, for each depth value of the original image, the depth value can be compared with the target depth threshold, and according to the comparison result, it can be determined whether the depth value corresponds to the background part, that is, whether the depth value is the depth value corresponding to the background part. If it is the depth value corresponding to the background part, the depth value needs to be filtered. By filtering out the depth value corresponding to the background part of the original image, the background part in the original image can be removed to obtain a process image.

[0086] For example, the original image is a film or television image, and the main objects such as the protagonist or important supporting roles are closer to the camera, while the other minor characters are farther away from the camera and serve as the background, so the background part of the original image needs to be removed.

[0087] Optionally, if it is determined according to the image encoding method that the larger the depth value, the more likely it is to correspond to the background part, then for each depth value of the original image, if the depth value is greater than or equal to the target depth threshold, it can be determined that the depth value corresponds to the background part and filtering processing is required.

[0088] Optionally, if it is determined according to the image encoding method that the smaller the depth value, the greater the possibility that it corresponds to the background part, then for each depth value of the original image, if the depth value is less than or equal to the target depth threshold, it can be determined that the depth value corresponds to the background part and filtering processing is required.

[0089] After obtaining the process image, target detection can be performed on the process image. Optionally, a target detection algorithm, such as yolov8, can be used to perform target detection on the process image. After performing target detection on the process image, an object detection frame in the process image can be obtained. For example, Figure 3 As shown, 310 represents a process image, 320 represents an object detection frame, and the specific content included in the process image is not shown.

[0090] If the number of object detection boxes in the process image is greater than or equal to the number of main objects, the process image may be determined as a target image.

[0091] First, the depth of the original image is estimated, and then the depth value of the original image is filtered based on the target depth threshold to remove the background part of the original image. When the number of object detection boxes in the obtained process image is greater than or equal to the number of main objects, the process image is determined as the target image. The target image that meets the conditions can be accurately obtained to ensure the smooth progress of subsequent steps.

[0092] In some embodiments of the present application, when the number of object detection boxes in the process image is less than or equal to the number of main objects, the method may further include the following steps:

[0093] Adjust the target depth threshold;

[0094] Repeat the step of comparing each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image.

[0095] In the embodiment of the present application, if the number of object detection frames in the process image is less than the number of main objects, it is considered that too much background has been removed from the original image, and the number of object detection frames in the process image is insufficient to match the number of main objects. In this case, the target depth threshold may be adjusted, and the step of comparing each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image may be repeated.

[0096] The purpose of adjusting the target depth threshold is to remove less background from the original image based on the target depth threshold, so that the number of object detection frames in the process image can be larger to match the number of main objects as soon as possible. Repeat the step of comparing each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background of the original image, where the target depth threshold refers to the adjusted target depth threshold.

[0097] That is, after determining the original image to be processed, firstly perform depth estimation on the original image to determine the depth value of the original image;

[0098] determining an initial depth threshold as a target depth threshold;

[0099] Compare each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image;

[0100] Filter out the depth value corresponding to the background part of the original image to remove the background part in the original image and obtain a process image;

[0101] Performing target detection on the process image to obtain the object detection frame in the process image;

[0102] If the number of object detection boxes in the process image is greater than or equal to the number of main objects, the process image is determined as a target image;

[0103] If the number of object detection boxes in the process image is less than the number of main objects, the object depth threshold is adjusted;

[0104] Compare each depth value of the original image with the adjusted target depth threshold to determine the depth value corresponding to the background portion of the original image;

[0105] Filter out the depth value corresponding to the background part of the original image to remove the background part in the original image and obtain a new process image;

[0106] Perform target detection on the new process image to obtain the object detection box in the image;

[0107] If the number of object detection boxes in the new process image is greater than or equal to the number of main objects, the new process image is determined as a target image;

[0108] If the number of object detection boxes in the new process image is less than the number of main objects, the target depth threshold is adjusted again and the related steps are repeated.

[0109] The above process may be performed multiple times until the number of object detection boxes in the process image is greater than or equal to the number of main objects.

[0110] This ensures that the number of object detection boxes in the target image is greater than or equal to the number of main objects, improving object recognition completeness.

[0111] In some embodiments of the present application, adjusting the target depth threshold may include the following steps:

[0112] If the depth value corresponding to the background part in the original image is less than the depth value corresponding to the foreground part, the target depth threshold is reduced according to the preset first step length;

[0113] If the depth value corresponding to the background portion in the original image is greater than the depth value corresponding to the foreground portion, the target depth threshold is increased according to a preset second step length.

[0114] In an embodiment of the present application, the purpose of adjusting the target depth threshold is to remove less background from the original image based on the target depth threshold, so that the number of object detection boxes in the process image can be larger to match the number of main objects as quickly as possible.

[0115] According to the image encoding method, if the depth value corresponding to the background part of the original image is smaller than the depth value corresponding to the foreground part, that is, the smaller the depth value, the greater the possibility of corresponding to the background part, then the target depth threshold can be reduced according to the first step. In this way, based on the reduced target depth threshold, the depth value of the original image is filtered, which will remove less background from the original image, and will also make the process image obtained by removing the background part have more object detection boxes.

[0116] According to the image encoding method, if the depth value corresponding to the background part in the original image is greater than the depth value corresponding to the foreground part, that is, the larger the depth value, the greater the possibility of corresponding to the background part, then the target depth threshold can be increased according to the second step length. In this way, based on the increased target depth threshold, the depth value of the original image is filtered, which will remove less background from the original image, and will also make the process image obtained by removing the background part have more object detection boxes.

[0117] The first step length and the second step length can be set and adjusted according to actual conditions, and the first step length and the second step length can be the same or different.

[0118] The object depth threshold is adjusted so that the number of object detection boxes in the process image can match the number of main objects as quickly as possible.

[0119] In some other embodiments of the present application, step S120 removes the background portion of the original image according to the depth value of the original image to obtain the target image, which may include the following steps:

[0120] Perform target detection on the original image to obtain the object detection frame in the original image;

[0121] When the number of object detection frames in the original image is greater than or equal to the number of main objects, the background portion of the original image is removed according to the depth value of the original image to obtain the target image.

[0122] In an embodiment of the present application, after determining the original image to be processed, target detection can be performed on the original image to obtain the object detection frame in the original image. And compare the number of object detection frames in the original image with the number of main objects. It can be understood that the original image is an image generated by the AI ​​model, and the image generation quality may not be stable. If the number of object detection frames in the original image is less than the number of main objects, it can be considered that the quality of such original image is poor, and it can be directly discarded or regenerated using the AI ​​model without subsequent image processing to save processing resources. If the number of object detection frames in the original image is greater than or equal to the number of main objects, it can be considered that the quality of such original image is good, and the subsequent steps can be continued, such as removing the background part of the original image according to the depth value of the original image, obtaining the target image and other subsequent steps.

[0123] After determining the original image to be processed, target detection is first performed on the original image. Only when the number of object detection frames in the original image is greater than or equal to the number of main objects, the subsequent steps are performed, which helps to improve the accuracy and completeness of object recognition.

[0124] In some embodiments of the present application, step S140 screens the object images according to the feature information corresponding to each object image, and may include the following steps:

[0125] Determine feature information corresponding to each object image, wherein the feature information corresponding to each object image includes image area and object posture;

[0126] Let i = 1, and filter the object images corresponding to the i-th object posture in the order of image area from large to small;

[0127] If the number of screened object images is greater than or equal to the number of primary objects, the screening is stopped;

[0128] If the number of screened object images is less than the number of main objects, let i=i+1, and repeat the step of screening the object images corresponding to the i-th object posture in the order of image area from large to small;

[0129] Among them, the priority of the object image corresponding to the i-th object posture is higher than the priority of the object image corresponding to the (i+1)-th object posture.

[0130] In an embodiment of the present application, after semantic segmentation is performed on the target image to obtain multiple object images, feature information corresponding to each object image can be further determined. The feature information corresponding to each object image may include image area and object posture.

[0131] Among them, for each object image, the image area of ​​the object image can be calculated by the coordinates of the mask points obtained by the semantic segmentation in step S130, and the object posture of the object image can be represented by a certain posture of the object, such as by the facial orientation. Taking the object as a person as an example, face recognition and posture recognition can be performed on each object image respectively, such as using the dwpose algorithm (an algorithm for human posture estimation) to parse the facial feature key points in each object image, and determine the facial orientation of each object. According to the orientation, it can be divided into three object postures: front face, side face / half face, and back.

[0132] Let i = 1, and the object images corresponding to the i-th object posture can be screened in order of image area from large to small. For each object image, if the image area of ​​the object image is larger, it is considered that the object contained in the object image is more likely to be the main object. Screening in order of image area from large to small helps to prioritize the main object.

[0133] If the number of the screened object images is greater than or equal to the number of the main objects, the screening may be stopped, and the currently screened object images may be determined as the last screened object images.

[0134] If the number of screened object images is less than the number of main objects, i=i+1 can be set, and the step of screening the object images corresponding to the i-th object posture in order of image area from large to small can be repeated.

[0135] That is, after determining the feature information corresponding to each object image, let i=1, and filter the object images corresponding to the first object posture in order from large to small image area. When the number of filtered object images is greater than or equal to the number of main objects, stop filtering.

[0136] If the number of screened object images is less than the number of main objects, that is, the number of object images corresponding to the first object posture is less than the number of main objects, then let i=i+1, that is, i=2, and screen the object images corresponding to the second object posture in the order of image area from large to small, and stop screening when the number of screened object images is greater than or equal to the number of main objects. Here, the number of screened object images includes the sum of the number of object images screened from the object images corresponding to the first object posture and the number of object images screened from the object images corresponding to the second object posture.

[0137] If the number of screened object images is less than the number of main objects, that is, the sum of the number of object images corresponding to the first object posture and the number of object images corresponding to the second object posture is less than the number of main objects, then let i=i+1, that is, i=3, and screen the object images corresponding to the third object posture in order from large to small image area. The process is similar and will not be repeated here.

[0138] Among them, the priority of the object image corresponding to the i-th object posture is higher than the priority of the object image corresponding to the (i+1)-th object posture.

[0139] For example, taking a person as an example, different object images correspond to three object postures: front face, side face / half face, and back face. The front face is the first object posture, the side face / half face is the second object posture, and the back face is the third object posture. It can be considered that the object image corresponding to the front face posture has the highest priority, the object image corresponding to the side face / half face posture has the second priority, and the object image corresponding to the back face posture has the lowest priority. This is because if the object is facing away from the camera, it is considered that it is not very meaningful to redraw its face, but the image area of ​​the object image corresponding to this object posture may be large, so it is necessary to first screen from the object images with higher priorities.

[0140] Sort the object images corresponding to the three object postures from large to small according to the image area;

[0141] First, filter the object images corresponding to the front face of the object. When the number of filtered object images is greater than or equal to the number of main objects, stop. If it is less than the number of main objects, continue filtering. When the object images corresponding to the front face of the object are filtered out, go to the next step.

[0142] After all the object images corresponding to the front face of the object are selected in the previous step, if the number of filtered object images is less than the number of main objects, the object images corresponding to the side face / half face of the object are filtered until the number of filtered object images is greater than or equal to the number of main objects. Similarly, if it is still not greater than or equal to the number of main objects, the object images corresponding to the back face of the object are filtered until the number of filtered object images is greater than or equal to the number of main objects.

[0143] Screening according to the image area and the object posture of each object image can improve the accuracy of the screened object images.

[0144] In some embodiments of the present application, after filtering the object images according to the feature information corresponding to each object image in step S140, the method further includes:

[0145] Adjusting other parts of the filtered object image except the object to a preset color;

[0146] The screened object images are merged into one image according to a preset picture sequence, and there is a preset interval between each object image in the merged image.

[0147] In the embodiment of the present application, after the object image is screened out, the other parts of the screened object image except the object can be adjusted to a preset color, such as white, to reduce the interference of non-objects contained in the object image on the object.

[0148] According to the preset picture order, the filtered object images can be merged into one image, and there is a preset interval between each object image in the merged image. In this way, the main object in the original image can be separated from the original image to generate a new image. Optionally, the merged image can be provided to the large model to provide more accurate input for the large model, facilitating the large model to accurately recognize the object.

[0149] The preset picture order may be the corresponding order of the object images in the original image. Optionally, the filtered object images may be sorted according to the size of the smallest x-coordinate of the bounding box and merged into one image.

[0150] For example, the merged image is Figure 4 As shown, 400 represents a merged image, 410, 420, 430 represent an object image respectively, and there is a preset interval between 410 and 420, and between 420 and 430. The content of the object image is not shown.

[0151] In some embodiments of the present application, the number of main objects in the original image can be determined by the following steps:

[0152] Get the prompt information of the original image;

[0153] According to the prompt information, determine the number of main objects in the original image.

[0154] In the embodiment of the present application, the original image may correspond to prompt information, and the prompt information of the original image may include a scene description and a description of each main object, and the prompt information meets a preset description specification. For example, the description of each main object is enclosed in square brackets, and the description of each main object may include a relevant description, code, etc. of the main object.

[0155] For example, the prompt information of the original image is as follows:

[0156] Three people sat together, laughing constantly. The background was a flickering light, full of a warm atmosphere. Wang XX raised his glass, "Li XX, Liu XX, thank you for organizing this party, giving us the opportunity to be together again." Li XX also raised his glass, "Cheers to our friendship!" Liu XX smiled and echoed, "I hope our friendship will last forever." [Young male, medium-length hair, left side of the screen, black suit, holding up a glass, Wang XX], [Young male, short hair, right side of the screen, white shirt, looking at the screen, holding up a glass, Li XX], [Young female, long hair, right side of the screen, holding up a glass, profile, Liu XX], Character information: Li XX, Liu XX, Wang XX.

[0157] It can be seen that the main objects contained in the original image are Li XX, Liu XX, and Wang XX. The first bracket encloses the description of Wang XX, the second bracket encloses the description of Li XX, and the third bracket encloses the description of Liu XX.

[0158] According to the prompt information, the number of main objects in the original image can be determined. Optionally, the number of main objects in the original image can be determined using a regular matching method. For example, the number of main objects in the original image can be determined by matching the square brackets in the prompt information using a regular matching method.

[0159] According to the prompt information of the original image, the number of main objects in the original image can be accurately determined.

[0160] The image processing solution provided in the embodiment of the present application can provide relatively accurate input for large models, help improve the accuracy and efficiency of large models in recognizing and processing images in complex scenes, and provide a guarantee for subsequent consistent drawing of objects.

[0161] Corresponding to the above method embodiment, the embodiment of the present application further provides an image processing device. The image processing device described below and the image processing method described above can refer to each other.

[0162] See also Figure 5 As shown, the image processing device 500 may include the following modules:

[0163] A determination module 510, used to determine an original image to be processed;

[0164] A first obtaining module 520 is used to remove the background part of the original image according to the depth value of the original image to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to the predetermined number of main objects in the original image;

[0165] A second obtaining module 530 is used to perform semantic segmentation on the target image to obtain a plurality of object images, each of which contains an object;

[0166] The screening module 540 is used to screen the object images according to the feature information corresponding to each object image, and the number of the screened object images is greater than or equal to the number of main objects, so as to perform object recognition based on the screened object images.

[0167] By using the device provided in the embodiment of the present application, after obtaining the original image to be processed, the background part in the original image is removed according to the depth value of the original image to obtain the target image, and the target image is semantically segmented to obtain multiple object images, each of which contains an object, and the object images are screened according to the feature information corresponding to each object image. That is, the original image is first processed to screen out the object images, and each screened object image contains an object, so that object recognition can be performed based on the screened object images. At the same time, the number of screened object images is greater than or equal to the number of main objects, so that the recognized objects can be better matched with the main objects in the original image. In addition, the screened object images do not contain the background part of the original image, which reduces the interference of the background part in the scene content expressed by the original image on the main object, so that when performing object recognition, more attention can be paid to analyzing and understanding the main object in the original image, which can improve the probability of correct recognition of objects in the image and improve recognition efficiency.

[0168] In some embodiments of the present application, the first obtaining module 520 is specifically used to:

[0169] Performing depth estimation on the original image to determine the depth value of the original image;

[0170] determining an initial depth threshold as a target depth threshold;

[0171] Compare each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image;

[0172] Filter out the depth value corresponding to the background part of the original image to remove the background part in the original image and obtain a process image;

[0173] Performing target detection on the process image to obtain the object detection frame in the process image;

[0174] In the case where the number of object detection frames in the process image is greater than or equal to the number of main objects, the process image is determined as a target image.

[0175] In some embodiments of the present application, the first obtaining module 520 is further used to:

[0176] In the case where the number of object detection boxes in the process image is less than the number of main objects, the target depth threshold is adjusted;

[0177] Repeat the step of comparing each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image.

[0178] In some embodiments of the present application, the first obtaining module 520 is specifically used to:

[0179] When the depth value corresponding to the background part in the original image is less than the depth value corresponding to the foreground part, the target depth threshold is reduced according to the preset first step length;

[0180] When the depth value corresponding to the background portion in the original image is greater than the depth value corresponding to the foreground portion, the target depth threshold is increased according to a preset second step size.

[0181] In some embodiments of the present application, the first obtaining module 520 is specifically used to:

[0182] Perform target detection on the original image to obtain the object detection frame in the original image;

[0183] When the number of object detection frames in the original image is greater than or equal to the number of main objects, the background portion of the original image is removed according to the depth value of the original image to obtain the target image.

[0184] In some embodiments of the present application, the screening module 540 is specifically used to:

[0185] Determine feature information corresponding to each object image, wherein the feature information corresponding to each object image includes image area and object posture;

[0186] Let i = 1, and filter the object images corresponding to the i-th object posture in the order of image area from large to small;

[0187] If the number of screened object images is greater than or equal to the number of primary objects, the screening is stopped;

[0188] If the number of screened object images is less than the number of main objects, let i=i+1, and repeat the step of screening the object images corresponding to the i-th object posture in the order of image area from large to small;

[0189] Among them, the priority of the object image corresponding to the i-th object posture is higher than the priority of the object image corresponding to the (i+1)-th object posture.

[0190] In some embodiments of the present application, the image processing apparatus 500 further includes a merging module, which is used to:

[0191] After filtering the object images according to the feature information corresponding to each object image, adjusting other parts of the filtered object images except the objects to a preset color;

[0192] The screened object images are merged into one image according to a preset picture sequence, and there is a preset interval between each object image in the merged image.

[0193] In some embodiments of the present application, the determination module 510 is further configured to determine the number of main objects in the original image through the following steps:

[0194] Get the prompt information of the original image;

[0195] According to the prompt information, determine the number of main objects in the original image.

[0196] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] The present application also provides an electronic device, such as Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604;

[0198] Memory 603, used for storing computer programs;

[0199] The processor 601 is used to execute the program stored in the memory 603 to implement the following steps:

[0200] Determine the original image to be processed;

[0201] According to the depth value of the original image, the background part of the original image is removed to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to the predetermined number of main objects in the original image;

[0202] Perform semantic segmentation on the target image to obtain multiple object images, each of which contains an object;

[0203] According to the feature information corresponding to each object image, the object images are screened, and the number of the screened object images is greater than or equal to the number of main objects, so as to perform object recognition based on the screened object images.

[0204] The communication bus 604 mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 604 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0205] The communication interface 602 is used for communication between the terminal and other devices.

[0206] The memory 603 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory 603 may also be at least one storage device located away from the aforementioned processor.

[0207] The above-mentioned processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0208] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which instructions are stored. When the computer-readable storage medium is executed on a computer, the computer executes the steps of any one of the image processing methods in the above embodiments.

[0209] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the steps of any one of the image processing methods in the above embodiments.

[0210] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loading and executing computer program instructions on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, a data center, etc. that contains one or more available media integrated. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0211] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0212] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0213] The above are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: include: Determine the original image to be processed; According to the depth value of the original image, a background portion of the original image is removed to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to a predetermined number of main objects in the original image; Performing semantic segmentation on the target image to obtain a plurality of object images, each of which contains an object; According to the feature information corresponding to each object image, the object images are screened, and the number of the screened object images is greater than or equal to the number of the main objects, so as to perform object recognition based on the screened object images.

2. The method according to claim 1, characterized in that The removing the background portion of the original image according to the depth value of the original image to obtain the target image comprises: Performing depth estimation on the original image to determine a depth value of the original image; determining an initial depth threshold as a target depth threshold; Compare each depth value of the original image with the target depth threshold to determine a depth value corresponding to a background portion of the original image; Filtering out the depth value corresponding to the background part of the original image to remove the background part in the original image to obtain a process image; Performing target detection on the process image to obtain an object detection frame in the process image; In a case where the number of object detection frames in the process image is greater than or equal to the number of main objects, the process image is determined as a target image.

3. The method according to claim 2, characterized in that In the case where the number of object detection frames in the process image is less than the number of main objects, the method further comprises: Adjusting the target depth threshold; Repeat the step of comparing each depth value of the original image with the target depth threshold to determine the depth value corresponding to the background portion of the original image.

4. The method according to claim 3, characterized in that The step of adjusting the target depth threshold comprises: If the depth value corresponding to the background part in the original image is less than the depth value corresponding to the foreground part, reducing the target depth threshold according to a preset first step length; If the depth value corresponding to the background portion in the original image is greater than the depth value corresponding to the foreground portion, the target depth threshold is increased according to a preset second step size.

5. The method according to claim 1, characterized in that The removing the background portion of the original image according to the depth value of the original image to obtain the target image comprises: Performing target detection on the original image to obtain an object detection frame in the original image; When the number of object detection frames in the original image is greater than or equal to the number of main objects, the background portion of the original image is removed according to the depth value of the original image to obtain a target image.

6. The method according to claim 1, characterized in that The step of screening the object images according to the feature information corresponding to each object image includes: Determine feature information corresponding to each object image, wherein the feature information corresponding to each object image includes image area and object posture; Let i = 1, and filter the object images corresponding to the i-th object posture in the order of image area from large to small; If the number of the screened object images is greater than or equal to the number of the main objects, stop screening; If the number of the screened object images is less than the number of the main objects, let i=i+1, and repeat the step of screening the object images corresponding to the i-th object posture in the order of image area from large to small; Among them, the priority of the object image corresponding to the i-th object posture is higher than the priority of the object image corresponding to the (i+1)-th object posture.

7. The method according to claim 1, characterized in that After filtering the object images according to the feature information corresponding to each object image, the method further includes: Adjusting other parts of the filtered object image except the object to a preset color; The screened object images are merged into one image according to a preset picture sequence, and there is a preset interval between each object image in the merged image.

8. The method according to any one of claims 1 to 7, characterized in that The number of main objects in the original image is determined by the following steps: Obtaining prompt information of the original image; According to the prompt information, the number of main objects in the original image is determined.

9. An image processing device, characterized in that: include: A determination module, used for determining an original image to be processed; A first obtaining module is used to remove the background part of the original image according to the depth value of the original image to obtain a target image, wherein the number of object detection frames in the target image is greater than or equal to the predetermined number of main objects in the original image; A second acquisition module is used to perform semantic segmentation on the target image to obtain a plurality of object images, each of which contains an object; The screening module is used to screen the object images according to the feature information corresponding to each object image, and the number of the screened object images is greater than or equal to the number of the main objects, so as to perform object recognition based on the screened object images.

10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of the image processing method according to any one of claims 1 to 8 when executing a program stored in a memory.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising computer instructions, which are stored in a computer-readable storage medium and are suitable for being read and executed by a processor, so that an electronic device having the processor performs the steps of the image processing method as described in any one of claims 1 to 8.

Images