Image processing method and device, storage medium and program product

Abstract

The invention discloses an image processing method and device, a storage medium and a program product, and relates to the technical field of image processing, and the image processing method comprises the steps: obtaining an original image, a semantic graph of the original image, and modification content description information for the original image; according to the semantic graph, identifying a target modification area where a target object pointed by the modification content description information is located; covering and redrawing a target modification area of the original image based on the semantic graph to obtain a background image; and fusing the target object with the background image to obtain a target image for constructing a world model. The technical problem that an existing world model construction method is difficult to realize adjustment of a single object is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, storage medium and program product. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the in-depth exploration in the field of autonomous driving, the world model, as an emerging and powerful tool, is gradually showing its great potential in understanding and simulating the real world.

[0003] However, the current way of building world models mainly relies on full-graph reasoning, which can capture the overall structure and relationship of an image or scene, but has obvious shortcomings when dealing with single objects. Since full-graph reasoning considers multiple objects and objects and backgrounds as one, it makes it extremely difficult to adjust a single object. In other words, the existing world model construction method is difficult to achieve the adjustment of a single object.

[0004] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of this application is to provide an image processing method, device, storage medium and program product, aiming to solve the technical problem that the existing world model construction method is difficult to achieve the adjustment of a single object.

[0006] To achieve the above object, the present application proposes an image processing method, which includes:

[0007] Acquire an original image, a semantic graph of the original image, and modified content description information for the original image;

[0008] According to the semantic graph, identifying a target modification area where the target object pointed to by the modification content description information is located;

[0009] Covering and redrawing the target modification area of ​​the original image based on the semantic graph to obtain a background image;

[0010] After the target object is fused with the background image, a target image is obtained, which is used to construct a world model.

[0011] In one embodiment, before the step of obtaining the semantic map of the original image, the following steps are included:

[0012] Extracting semantics from each pixel in the original image to obtain pixel representation semantics;

[0013] Performing semantic extraction on features of each label category in the original image to obtain label feature semantics;

[0014] A semantic map of the original image is generated according to the pixel representation semantics and the label feature semantics.

[0015] In one embodiment, the step of overwriting and redrawing the target modification area of ​​the original image based on the semantic graph to obtain a background image includes:

[0016] Acquire background information of the original image, and generate an initial background according to the background information;

[0017] The target modification area of ​​the original image is covered, and the covered area is redrawn based on the semantic map and the initial background to generate a background image corresponding to the original image.

[0018] In one embodiment, the step of covering the target modification area of ​​the original image and redrawing the covered area based on the semantic map and the initial background to generate a background image corresponding to the original image includes:

[0019] Performing covering and noise processing on the target modification area of ​​the original image, and deleting the object in the covering area from the original image after the noise processing based on the semantic graph to obtain a first image;

[0020] Denoising the initial background, and deleting objects outside the target modification area from the initial background based on the semantic graph to obtain a second image;

[0021] fusing the first image and the second image to obtain a current modified image;

[0022] The current modified image is used as a new initial background, and the steps are performed: covering and denoising the target modified area of ​​the original image, and deleting the objects in the covered area from the original image after the denoising based on the semantic graph to obtain a first image;

[0023] After a predetermined iteration time, the current modified image is used as the background image.

[0024] In one embodiment, the step of fusing the target object with the background image to obtain the target image comprises:

[0025] Performing text extraction on the modified content description information to obtain a location modification instruction corresponding to the modified content description information, and determining a target location indicated by the location modification instruction;

[0026] After the target object is moved to the target position, the moved target object is fused with the background image to obtain a target image.

[0027] In one embodiment, after the target object is moved to the target position, before the step of fusing the moved target object with the background image to obtain the target image, the method includes:

[0028] Obtaining an initial position of the target object in the original image;

[0029] Determine, according to the initial position and the target position, a first mapping ratio of the initial position in a world coordinate system and a second mapping ratio of the target position in the world coordinate system;

[0030] The size of the target object is adjusted according to the first mapping ratio and the second mapping ratio to obtain an adjusted target object.

[0031] In one embodiment, after the step of fusing the target object with the background image according to the modification instruction corresponding to the modification content description information to obtain the target image, the method further includes:

[0032] Acquire an image sequence and prior optical flow information of a camera that captures the original image, wherein the image sequence includes a target image;

[0033] According to the target image, generating three-dimensional layout information of the target image;

[0034] Performing optical flow estimation on the image sequence based on the prior optical flow information to obtain dense optical flow between adjacent frames in the image sequence;

[0035] A scene video is generated according to the dense optical flow, the three-dimensional layout information and the image sequence for constructing a world model.

[0036] In addition, to achieve the above objectives, the present application also proposes an image processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image processing method described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image processing method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the image processing method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] The present application obtains the original image, the semantic graph of the original image, and the modified content description information for the original image. Thus, the original image and its semantic graph expected to be used to construct the world model are obtained, as well as the text content of the modification requirements for the original image issued by the user. Then, according to the semantic graph, the target modification area where the target object pointed to by the modified content description information is located is identified. Since the semantic graph will segment each object in the original image, the segmented area is matched with the target object pointed to by the modified content description information, and the target modification area that needs to be modified can be identified. Based on the semantic graph, the target modification area of ​​the original image is covered and redrawn to obtain a background image. Since the semantic graph describes the characteristics of each object in the original image, the semantic graph is covered and redrawn, which can ensure that the morphology of each object in the original image will not be out of control or deformed during the image processing process. Then, according to the modification instructions corresponding to the modified content description information (such as adding, deleting, adjusting the target object, etc.), the target object is fused with the background image to obtain a target image for constructing a world model. Therefore, on the one hand, the present application can accurately identify the target object that needs to be modified through the semantic graph of the original image, and then through the modification of the target object, the original image can be personalized edited and adjusted according to specific needs. On the other hand, by redrawing and fusion under the guidance of the semantic graph, the morphology of each object in the original image can be constrained to avoid morphological deformation and loss of control during the image editing and adjustment process, resulting in the target image obtained after adjustment being unable to be used to construct the world model. Therefore, the present application can edit and adjust the image of a single object on the premise that the object morphology is controllable, and can realize the adjustment of a single object in the constructed world model, effectively improving the scope of application and construction flexibility of the world model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the image processing method of the present application;

[0044] Figure 2 A scene graph for generating a semantic graph involved in the embodiment of the present application;

[0045] Figure 3 This is an example diagram of image processing without semantic graph constraints involved in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of image processing under semantic graph constraints involved in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a scene for generating a scene video according to an embodiment of the present application;

[0048] Figure 6 A schematic diagram of a flow chart provided for the second embodiment of the image processing method of the present application;

[0049] Figure 7 A scene graph for generating a target image involved in the embodiment of the present application;

[0050] Figure 8 A schematic diagram of a process flow provided for the third embodiment of the image processing method of the present application;

[0051] Fig. 9 Schematic diagram of the device structure of the hardware operating environment involved in the image processing method in the embodiment of the present application.

[0052] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtaining an original image, a semantic map of the original image, and modified content description information for the original image; identifying a target modification area where a target object pointed to by the modified content description information is located based on the semantic map; overwriting and redrawing the target modification area of ​​the original image based on the semantic map to obtain a background image; and fusing the target object with the background image to obtain a target image for constructing a world model.

[0056] In this embodiment, for the convenience of description, the vehicle controller is used as the execution subject for explanation below.

[0057] With the rapid development of artificial intelligence technology, especially the in-depth exploration in the field of autonomous driving, the world model, as an emerging and powerful tool, is gradually showing its great potential in understanding and simulating the real world.

[0058] However, the current way of building world models mainly relies on full-graph reasoning, which can capture the overall structure and relationship of an image or scene, but has obvious shortcomings when dealing with single objects. Since full-graph reasoning considers multiple objects and objects and backgrounds as one, it makes it extremely difficult to adjust a single object. In other words, the existing world model construction method is difficult to achieve the adjustment of a single object.

[0059] The present application provides a solution. On the one hand, through the semantic graph of the original image, the target object that needs to be modified can be accurately identified, and then by modifying the target object, the original image can be personalized edited and adjusted according to specific needs. On the other hand, by redrawing and fusion under the guidance of the semantic graph, the morphology of each object in the original image can be constrained to avoid morphological deformation and loss of control during the image editing and adjustment process, resulting in the target image obtained after adjustment being unable to be used to construct a world model. Therefore, the present application can edit and adjust the image of a single object under the premise that the object morphology is controllable, and can realize the adjustment of a single object in the constructed world model, effectively improving the scope of application and construction flexibility of the world model.

[0060] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a vehicle controller, a server, etc. The following takes the vehicle controller as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the present application embodiment provides an image processing method, referring to Figure 1 , Figure 1 This is a flowchart diagram of the first embodiment of the image processing method of the present application.

[0062] In this embodiment, the image processing method includes steps S10 to S40:

[0063] Step S10, obtaining an original image, a semantic graph of the original image, and modified content description information for the original image;

[0064] It should be noted that the original image is the initial image used to construct the world model, and the semantic graph is a graph composed of the semantics corresponding to the features of each element (such as pixels, specific objects, etc.) in the original image. Exemplarily, the semantic graph at least includes the semantics corresponding to the morphological features of each object in the original image. The modified content description information is the original natural language information that describes the modified content of the original image, and of course it can also be a preprocessed text obtained by preprocessing the natural language information. It can be understood that the preprocessing can include data cleaning, text standardization, word segmentation, text vectorization, etc., which are used to convert the original, unstructured original natural language information into structured, numerical text data.

[0065] In this embodiment, the original image can be obtained, and the feature semantics of the original image can be obtained by semantically extracting the features of each element in the original image, and a semantic graph of the original image can be generated based on the feature semantics. In addition, the modified content description information can be obtained by receiving the modified content description information input by the user for the original image. Exemplarily, the modified content description information can be natural language such as "move the front vehicle to the right lane", "add a vehicle to the right lane", "delete the front vehicle", etc., or it can be a structured text input after preprocessing, such as "move the vehicle to the right side [150,80]" obtained after preprocessing.

[0066] In a feasible implementation manner, before the step of obtaining the semantic map of the original image in step S10, steps A10 to A40 may be included:

[0067] Step A10, performing semantic extraction on each pixel in the original image to obtain pixel representation semantics;

[0068] Step A20, performing semantic extraction on features of each label category in the original image to obtain label feature semantics;

[0069] Step A30: Generate a semantic map of the original image according to the pixel representation semantics and the label feature semantics.

[0070] In order to accurately simulate and reproduce the complexity and diversity of the real world in the world model, it provides strong support for subsequent visual tasks (such as image generation, scene understanding, etc.). Figure 2 , Figure 2The scene graph for generating the semantic graph involved in the embodiment of the present application is shown in FIG. This embodiment adopts a multi-level and multi-dimensional information extraction strategy for the semantic graph of each original image. The present application first performs semantic segmentation on the original image to obtain the semantic objects and their label categories corresponding to the semantic segmentation area. Then, at the pixel level, semantic extraction is performed on each pixel in the original image to obtain pixel representation semantics. The pixel representation semantics is the semantic content represented by each pixel in the original image. For example, a certain pixel may represent the head of a person or the headlight of a car. At the fine granularity of the pixel level, the microscopic structure of the original image can be effectively and deeply identified. Furthermore, at the label category level, this embodiment performs semantic extraction on the features of each label category in the original image to obtain label feature semantics. This embodiment extracts the semantic content of the overall features of each label category for each label category (such as people, vehicles, buildings, etc.). The overall features of each label category describe the commonalities between different objects in the same label category. Therefore, at the fine granularity of the label category, the roles and relationships of different objects in the overall scene can be effectively identified. Furthermore, as Figure 2 As shown, this embodiment can generate corresponding semantic representation graphs according to the pixel representation semantics and the label feature semantics, and then use the image after the fusion of the semantic representation graphs as the semantic graph of the original image. Based on the semantic graph, this embodiment can simultaneously identify the layout of each element in the entire original image and the morphological features of a single label category object, so that in the subsequent image processing process, the shape of each object in the original image is effectively constrained and more similar to the real object. Figure 3 , Figure 3 : is an example of image processing without semantic graph constraints involved in the embodiment of the present application. The image after image processing without semantic graph constraints is as follows Figure 3 As shown, Figure 3 The shapes of the people in the middle dashed box are obviously out of control. Such images cannot be used to construct a world model, which will cause serious distortion of the world model. Figure 4 , Figure 4 A schematic diagram of image processing under semantic graph constraints involved in an embodiment of the present application. Figure 4 The two leftmost images are semantic images, the two middle images are images before processing, and the two rightmost images are images after processing. Figure 4 As shown in Figure 2, under the constraints of the semantic graph, the shapes of people and objects in the dotted line frame do not change significantly in the images before and after processing.

[0071] This embodiment performs semantic extraction on each pixel in the original image to obtain pixel representation semantics; performs semantic extraction on the features of each label category in the original image to obtain label feature semantics; performs semantic extraction on each pixel in the original image combined with each label category to obtain fusion semantics; generates a semantic map of the original image based on the pixel representation semantics and the label feature semantics. This embodiment generates a semantic map of the original image by adopting a multi-level and multi-dimensional information extraction strategy, thereby maintaining the controllable morphology of each object in the original image during the image processing process, and the rich semantic information also provides strong support for the subsequent accurate simulation and reproduction of the complexity and diversity of the real world in the world model.

[0072] Step S20, identifying the target modification area where the target object pointed to by the modification content description information is located according to the semantic graph;

[0073] It should be noted that the target object is the object in the original image that is pointed to and desired to be modified in the modification content description information.

[0074] In this embodiment, the modified object in the modified content description information can be obtained by performing text recognition on the modified content description information. Then, the modified object is matched with the characteristic semantics of each semantic object in the semantic graph, the matched semantic object is used as the target object, and the semantic segmentation area where the target object is located is used as the target modification area. Exemplarily, if the modified object in the modified content description information is the front vehicle, the semantic graph can be traversed to identify the target vehicle that matches the front vehicle in the semantic graph, and the semantic segmentation area of ​​the target vehicle segmented in the semantic segmentation process is used as the target modification area.

[0075] Step S30, covering and redrawing the target modification area of ​​the original image based on the semantic graph to obtain a background image;

[0076] In this embodiment, the target modification area of ​​the original image can be covered based on the semantic map to mark the covered area for segmentation, and then the pre-trained image segmentation model (such as U-Net, DeepLab, etc.) can be used to segment the covered area and retain the background part. Then, by filling and redrawing the covered area, a background image in which the target object is deleted from the original image is obtained. Exemplarily, the method of filling and redrawing the covered area can be a method using a trained neural network model (such as a generative adversarial network), an inpaint function in a library such as OpenCV, or a similar function. The above method can fill and redraw the covered area based on the surrounding pixel information to simulate the effect of deleting the target object from the original image.

[0077] Step S40: After fusing the target object with the background image, a target image is obtained for constructing a world model.

[0078] This embodiment can perform text extraction through the modified content description information to obtain corresponding modification instructions, such as position modification instructions, object addition instructions, object deletion instructions, etc. Then, the target object is fused with the background image according to the modification instructions to obtain a target image. Then, this embodiment can construct a corresponding world model based on the target image.

[0079] As an example, this embodiment can perform text extraction on the modified content description information, obtain the position modification instruction corresponding to the modified content description information, and determine the target position indicated by the position modification instruction; after moving the target object to the target position, the moved target object is merged with the background image to obtain the target image.

[0080] As another example, this embodiment can perform text extraction on the modified content description information to obtain a delete object instruction corresponding to the modified content description information, and then directly use the background image as the target image after deleting the target object according to the delete object instruction. As another example, this embodiment can perform text extraction on the modified content description information to obtain an add object instruction corresponding to the modified content description information, and then add the target object according to the add object instruction, and then merge the added target object with the background image to obtain the target image.

[0081] In a feasible embodiment, step S40 includes steps S50 to S80:

[0082] Step S50, acquiring an image sequence and prior optical flow information of a camera that photographs the original image, wherein the image sequence includes a target image;

[0083] Step S60, generating three-dimensional layout information of the target image according to the target image;

[0084] Step S70, performing optical flow estimation on the image sequence based on the prior optical flow information to obtain dense optical flow between adjacent frames in the image sequence;

[0085] Step S80, generating a scene video according to the dense optical flow, the three-dimensional layout information and the image sequence for constructing a world model.

[0086] It should be noted that the prior optical flow information includes the optical flow of the target object in the original image obtained by predicting the captured image of the camera in advance. The image sequence may include the original image and at least one target image, or at least two target images as each frame image in the image sequence. Exemplarily, this embodiment requires the original image to be transformed into a world model of the scene under the target image, then the image sequence includes the original image and at least one target image. This embodiment requires the world model of the scene to be transformed between each of the target images, then the image sequence includes at least two target images.

[0087] In order to generate a world model in a dynamic scene, this embodiment can obtain an image sequence and a priori optical flow information of the camera that shoots the original image, wherein the image sequence includes a target image. Based on the target image, three-dimensional layout information of the target image is generated. The three-dimensional layout information includes information such as the external dimensions and spatial positions of each object in the target image. Exemplarily, this embodiment can query the mapping relationship between a preset pixel coordinate system and a world coordinate system based on the pixel coordinates of each object in the target image in the target image, and obtain three-dimensional layout information such as the external dimensions of each object and the spatial position in the world coordinate system. Then, based on the prior optical flow information, optical flow estimation is performed on the image sequence to obtain a dense optical flow between adjacent frames in the image sequence, wherein the dense optical flow is a dense optical flow field formed by the offset of all pixel points on each frame image in the image sequence. Finally, based on the dense optical flow, the three-dimensional layout information and the image sequence, each frame image in the image sequence is interpolated to generate a scene video for constructing a world model. See. Figure 5 , Figure 5 The present invention is a schematic diagram of a scene generation scenario of a scene video involved in an embodiment of the present application. In this embodiment, after the original image is input into the target image generation module in the image processing device, the target image generation module performs steps S10 to S40 based on the original image to obtain the target image. Then, an image sequence is obtained, and the prior optical flow information of the camera that shoots the original image, wherein the image sequence includes a target image; based on the target image, three-dimensional layout information of the target image is generated; based on the prior optical flow information, optical flow estimation is performed on the image sequence to obtain dense optical flow between adjacent frames in the image sequence; based on the dense optical flow, the three-dimensional layout information and the image sequence, a scene video is generated.

[0088] This embodiment obtains an image sequence and the prior optical flow information of the camera that captured the original image, and generates the three-dimensional layout information of the target image according to the target image; performs optical flow estimation on the image sequence based on the prior optical flow information to obtain dense optical flow between adjacent frames in the image sequence; and performs frame complementation on each frame in the image sequence according to the dense optical flow, the three-dimensional layout information and the image sequence to generate a scene video for constructing a world model. The world model constructed by this embodiment can show the real world scene of the transformation from the original image to the target image, or the real world scene of the transformation between the target images.

[0089] The first embodiment of the present application provides an image processing method. In this embodiment, an original image, a semantic graph of the original image, and modified content description information for the original image are obtained. Thus, the original image and its semantic graph expected to be used to construct a world model are obtained, as well as the text content of the modification requirements for the original image issued by the user. Then, according to the semantic graph, the target modification area where the target object pointed to by the modified content description information is located is identified. Since the semantic graph will segment each object in the original image, the segmented area is matched with the target object pointed to by the modified content description information, and the target modification area that needs to be modified can be identified. Based on the semantic graph, the target modification area of ​​the original image is covered and redrawn to obtain a background image. Since the semantic graph describes the characteristics of each object in the original image, the semantic graph is covered and redrawn, which can ensure that the morphology of each object in the original image will not be out of control or deformed during the image processing process. Then, according to the modification instructions corresponding to the modified content description information (such as adding, deleting, adjusting the target object, etc.), the target object is fused with the background image to obtain a target image for constructing a world model. Therefore, on the one hand, this embodiment can accurately identify the target object that needs to be modified through the semantic graph of the original image, and then through the modification of the target object, the original image can be personalized edited and adjusted according to specific needs. On the other hand, by redrawing and fusion under the guidance of the semantic graph, the morphology of each object in the original image can be constrained to avoid morphological deformation and loss of control during the image editing and adjustment process, resulting in the target image obtained after adjustment being unable to be used to construct the world model. Therefore, this embodiment can edit and adjust the image of a single object on the premise that the original image is controllable in object morphology, and can achieve adjustment of a single object in the constructed world model, effectively improving the scope of application and construction flexibility of the world model.

[0090] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 6 , step S30 includes steps S31-S32:

[0091] Step S31, obtaining background information of the original image, and generating an initial background according to the background information;

[0092] Step S32, covering the target modification area of ​​the original image, and redrawing the covered area based on the semantic map and the initial background to generate a background image corresponding to the original image.

[0093] It should be noted that the background information is the image information of the area other than the target object in the original image.

[0094] This embodiment can use a pre-trained background generation model, and after obtaining the background information of the original image, generate an initial background based on the background information. Then, the target modification area of ​​the original image is covered to identify the area that needs to be modified. Then, the covered area is redrawn based on the semantic map and the initial background to generate a background image corresponding to the original image. It can be understood that the method of redrawing the covered area can use the diffusion model, the generative adversarial network, the inpaint function in the OpenCV library or similar functions to fill and redraw the covered area based on the surrounding pixel information of the covered area. In addition, the method of covering and redrawing based on the initial background of the original image combined with the original image can also make the covered area more naturally connected with other areas after redrawing, so as to improve the authenticity of the background image corresponding to the original image.

[0095] In a feasible embodiment, step S32 includes steps B10 to B50:

[0096] Step B10, performing covering and noise processing on the target modification area of ​​the original image, and deleting the objects in the covering area from the original image after the noise processing based on the semantic graph to obtain a first image;

[0097] Step B20, performing denoising processing on the initial background, and deleting objects outside the target modification area from the initial background based on the semantic graph to obtain a second image;

[0098] Step B30, fusing the first image and the second image to obtain a current modified image;

[0099] Step B40, using the current modified image as a new initial background, and executing the steps of: covering and denoising the target modified area of ​​the original image, and deleting the objects in the covered area from the original image after the denoising based on the semantic graph, to obtain a first image;

[0100] Step B50: After a predetermined iteration time, the current modified image is used as the background image.

[0101] See also Figure 7 , Figure 7 The generated scene graph of the target image involved in the embodiment of the present application. This embodiment covers the target modification area of ​​the original image based on the semantic graph to identify the area to be modified (i.e. Figure 7 The method further comprises the following steps: first, performing a denoising process on the initial background to maintain the clarity and stability of the initial background. Second, deleting objects outside the target modification area from the initial background based on the semantic graph to obtain a second image. Since the first image does not contain objects within the target modification area, and the second image does not contain objects outside the target modification area, the second image can be overlapped with the first image to achieve the fusion of the first image and the second image to obtain the current modified image. Third, taking the current modified image as the new initial background, and executing the steps of covering and denoising the target modification area of ​​the original image, and deleting objects within the covered area from the original image after the denoising based on the semantic graph to obtain the first image. After a predetermined iteration time, taking the current modified image as the background image. Therefore, this embodiment continuously adds noise to the original image and the initial background to simulate the transition effect, and iterates the noise removal to improve the clarity, thereby achieving a good transition between the modified target area in the original image and other objects after modification, and effectively improving the authenticity of the background image. Figure 7 As shown, based on the modified content description information (ie Figure 7 The “car, right, car, [150, 80]” in the image means moving the car to the right position [150, 80]. Based on the semantic graph, the target object (i.e., the car) in the original image is fused with the background image of the original image, and finally the target image is output.

[0102] In the second embodiment of the present application, the background information of the original image is obtained, and an initial background is generated according to the background information; the target modification area of ​​the original image is covered, and the covered area is redrawn based on the semantic graph and the initial background to generate a background image corresponding to the original image. The method of overlaying and redrawing based on the initial background of the original image combined with the original image in this embodiment can also make the overlay area more naturally connected with other areas after redrawing, so as to improve the authenticity of the background image corresponding to the original image.

[0103] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 8 , step S40 includes steps S41 to S42:

[0104] Step S41, performing text extraction on the modified content description information to obtain a location modification instruction corresponding to the modified content description information, and determining a target location indicated by the location modification instruction;

[0105] Step S42, after the target object is moved to the target position, the moved target object is merged with the background image to obtain a target image.

[0106] The modification instruction may be at least one of an add object instruction (i.e., add the target object), a delete object instruction (i.e., delete the target object), and a position modification instruction (i.e., move the position of the target object) for the target object.

[0107] In this embodiment, taking the modification instruction as a position modification instruction as an example, this embodiment can perform text extraction on the modification content description information through a preset language model, obtain the position modification instruction corresponding to the modification content description information, and determine the target position indicated by the position modification instruction, and the target position is the position described in the modification content description information to which the target object is expected to move. Furthermore, this embodiment can move the target object to the target position, and then fuse the moved target object with the background image to obtain a target image. It should be noted that this embodiment can use image fusion methods such as weighted averaging, Laplace pyramid fusion, and multi-scale fusion to fuse the moved target object with the background image to obtain a target image. Of course, this embodiment can also perform post-processing (such as denoising, smoothing, sharpening, etc.) on the fused target image to improve the image quality of the target image.

[0108] In a feasible embodiment, the step S42 includes steps C10 to C30:

[0109] Step C10, obtaining the initial position of the target object in the original image;

[0110] Step C20, determining a first mapping ratio of the initial position in a world coordinate system and a second mapping ratio of the target position in the world coordinate system according to the initial position and the target position;

[0111] Step C30: adjusting the size of the target object according to the first mapping ratio and the second mapping ratio to obtain an adjusted target object.

[0112] It should be noted that the first mapping ratio is the ratio between the pixel distance of the object at the initial position in the original image and the real distance in the world coordinates. The second mapping ratio is the ratio between the pixel distance of the object at the target position in the original image and the real distance in the world coordinates.

[0113] During the movement of the target object, if it is far from the initial position, the size of the target object in the camera shooting picture will change greatly. Therefore, in order to ensure that the size of the target object in the world model constructed based on the target image is consistent with the real size. In this embodiment, the initial position of the target object in the original image can be obtained. Then, according to the initial position and the target position, the first mapping ratio of the initial position in the world coordinate system and the second mapping ratio of the target position in the world coordinate system are determined. In this embodiment, a mapping relationship between the shooting picture of the camera that shoots the original image and the world coordinate system can be established, and then two first reference points on the same horizontal line are selected from the initial position, and then two first real reference points in the world coordinate system are obtained by querying the mapping relationship. The ratio between the pixel distance between the two first reference points and the real distance between the two first real reference points is used as the first mapping ratio. Similarly, two second reference points on the same horizontal line are selected from the target position, and then two second real reference points in the world coordinate system are obtained by querying the mapping relationship. The ratio between the pixel distance between the two second reference points and the real distance between the two second real reference points is used as the second mapping ratio. Of course, the present embodiment can also establish a correspondence between the real distance between the object and the camera and the mapping ratio, and obtain the first mapping ratio of the initial position in the world coordinate system and the second mapping ratio of the target position in the world coordinate system by querying the corresponding relationship. The size scaling ratio of the target object is calculated according to the first mapping ratio and the second mapping ratio. Exemplarily, the first mapping ratio is the value of the pixel distance to the real distance of the initial position, and the second mapping ratio is the value of the pixel distance to the real distance of the target position, then the size scaling ratio of the target object is the second mapping ratio divided by the first mapping ratio. The first mapping ratio is the value of the real distance to the pixel distance of the initial position, and the second mapping ratio is the value of the real distance to the pixel distance of the target position, then the size scaling ratio of the target object is the first mapping ratio divided by the second mapping ratio. Then, the target object is scaled proportionally according to the size scaling ratio to achieve the size adjustment of the target object, and the adjusted target object is obtained.

[0114] This embodiment obtains the initial position of the target object in the original image; determines the first mapping ratio of the initial position in the world coordinate system and the second mapping ratio of the target position in the world coordinate system according to the initial position and the target position; and adjusts the size of the target object according to the first mapping ratio and the second mapping ratio to obtain the adjusted target object. This ensures that after the target object moves, the size of the target object in the world model constructed based on the target image can remain consistent with the actual size, effectively improving the authenticity of the world model.

[0115] The third embodiment of the present application extracts the text of the modified content description information to obtain the position modification instruction corresponding to the modified content description information, and determines the target position indicated by the position modification instruction; after moving the target object to the target position, the moved target object is merged with the background image to obtain the target image. Thus, this embodiment realizes the free movement of a single object in the original image, thereby realizing the free movement of a single object in the world model constructed based on the target image.

[0116] The present application provides an image processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image processing method in the above-mentioned embodiment one.

[0117] Reference below Fig. 9 , which shows a schematic diagram of the structure of an image processing device suitable for implementing an embodiment of the present application. The image processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The image processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0118] like Fig. 9As shown, the image processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the image processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the image processing device to communicate with other devices wirelessly or by wire to exchange data. Although the image processing device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0119] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0120] The image processing device provided by the present application adopts the image processing method in the above embodiment, which can solve the technical problem that the existing image processing method is difficult to adjust a single object. Compared with the prior art, the beneficial effects of the image processing device provided by the present application are the same as the beneficial effects of the image processing method provided by the above embodiment, and other technical features in the image processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0121] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0123] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the image processing method in the above-mentioned embodiment.

[0124] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0125] The computer-readable storage medium may be included in the image processing device; or may exist independently without being assembled into the image processing device.

[0126] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the image processing device, the image processing device: obtains an original image, a semantic graph of the original image, and modified content description information for the original image; identifies a target modification area where a target object pointed to by the modified content description information is located based on the semantic graph; overwrites and redraws the target modification area of ​​the original image based on the semantic graph to obtain a background image; and obtains a target image after fusing the target object with the background image for constructing a world model.

[0127] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0130] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned image processing method, and can solve the technical problem that the existing image processing method is difficult to adjust a single object. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the image processing method provided by the above-mentioned embodiment, and will not be repeated here.

[0131] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0132] The computer program product provided by the present application can solve the technical problem that the existing image processing method is difficult to adjust a single object. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the image processing method provided by the above embodiment, which will not be repeated here.

[0133] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. An image processing method, characterized in that: The image processing method comprises: Acquire an original image, a semantic graph of the original image, and modified content description information for the original image; According to the semantic graph, identifying a target modification area where the target object pointed to by the modification content description information is located; Covering and redrawing the target modification area of ​​the original image based on the semantic graph to obtain a background image; After the target object is fused with the background image, a target image is obtained, which is used to construct a world model.

2. The method according to claim 1, characterized in that Before the step of obtaining the semantic map of the original image, it includes: Extracting semantics from each pixel in the original image to obtain pixel representation semantics; Performing semantic extraction on features of each label category in the original image to obtain label feature semantics; A semantic map of the original image is generated according to the pixel representation semantics and the label feature semantics.

3. The method according to claim 1, characterized in that The step of covering and redrawing the target modification area of ​​the original image based on the semantic graph to obtain a background image includes: Acquire background information of the original image, and generate an initial background according to the background information; The target modification area of ​​the original image is covered, and the covered area is redrawn based on the semantic map and the initial background to generate a background image corresponding to the original image.

4. The method according to claim 3, characterized in that The step of covering the target modification area of ​​the original image and redrawing the covered area based on the semantic map and the initial background to generate a background image corresponding to the original image includes: Performing covering and noise processing on the target modification area of ​​the original image, and deleting the object in the covering area from the original image after the noise processing based on the semantic graph to obtain a first image; Denoising the initial background, and deleting objects outside the target modification area from the initial background based on the semantic graph to obtain a second image; fusing the first image and the second image to obtain a current modified image; The current modified image is used as a new initial background, and the steps are performed: covering and denoising the target modified area of ​​the original image, and deleting the objects in the covered area from the original image after the denoising based on the semantic graph to obtain a first image; After a predetermined iteration time, the current modified image is used as the background image.

5. The method according to claim 1, characterized in that The step of obtaining a target image after fusing the target object with the background image comprises: Performing text extraction on the modified content description information to obtain a location modification instruction corresponding to the modified content description information, and determining a target location indicated by the location modification instruction; After the target object is moved to the target position, the moved target object is fused with the background image to obtain a target image.

6. The method according to claim 5, characterized in that After the target object is moved to the target position, before the step of fusing the moved target object with the background image to obtain the target image, the method further comprises: Obtaining an initial position of the target object in the original image; Determine, according to the initial position and the target position, a first mapping ratio of the initial position in a world coordinate system and a second mapping ratio of the target position in the world coordinate system; The size of the target object is adjusted according to the first mapping ratio and the second mapping ratio to obtain an adjusted target object.

7. The method according to any one of claims 1 to 6, characterized in that After the step of fusing the target object with the background image according to the modification instruction corresponding to the modification content description information to obtain the target image, the method further comprises: Acquire an image sequence and prior optical flow information of a camera that captures the original image, wherein the image sequence includes a target image; According to the target image, generating three-dimensional layout information of the target image; Performing optical flow estimation on the image sequence based on the prior optical flow information to obtain dense optical flow between adjacent frames in the image sequence; A scene video is generated according to the dense optical flow, the three-dimensional layout information and the image sequence for constructing a world model.

8. An image processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the image processing method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images