Image processing method, model training method and related devices

Abstract

The embodiment of the invention discloses an image processing method, a model training method and a related device, at least relates to technologies such as artificial intelligence and the like, and is used for generating a target projection scene graph meeting conditions so as to reduce image deviation and realize rapid and efficient construction of a three-dimensional scene graph. The method comprises the steps of obtaining a to-be-processed image and text description information; performing regular equiaxed projection transformation processing on the to-be-processed image to obtain a target semantic segmentation map; performing diffusion processing on the text description information and the target semantic segmentation graph based on a target diffusion generation model to obtain a target projection scene graph, the target diffusion generation model being a machine learning model obtained by training an initial diffusion generation model by taking a sample projection scene graph and a label projection scene graph as training samples, the sample projection scene graph is obtained by performing diffusion processing on the sample description information and the sample semantic segmentation graph through the initial diffusion generation model; and generating a target three-dimensional scene graph based on the target projection scene graph and the marker.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of image processing, and in particular, to a method for image processing, a method for model training, and related devices. Background Art

[0002] With the rapid development of technologies such as artificial intelligence technology, graphics rendering technology, and image technology, it has gradually developed to use artificial intelligence technology and the like to generate images with diverse styles. For example, generate landscape pictures with rich content of urban themes, natural themes, etc.; or generate game scene pictures, etc.

[0003] In traditional image generation schemes, usually, the text description information used to describe the scene is used as the input of the diffusion model, so as to use the diffusion model as an image generation tool to generate an image that conforms to the text description information.

[0004] However, although the above-mentioned diffusion model can generate relevant images by processing the text description information of the scene. However, the traditional diffusion model is prone to generating multiple images with unfixed viewpoints, and each image can only meet the general text description. Or rather, using the traditional diffusion model is extremely likely to cause a deviation between the generated image and the image actually described by the text description information, and it cannot be used to generate a three-dimensional scene map, resulting in a poor construction efficiency of the three-dimensional scene. Summary of the Invention

[0005] The embodiments of the present application provide a method for image processing, a method for model training, and related devices, which are used to generate a target projection scene map that meets the conditions, so as to reduce image deviation and achieve rapid and efficient construction of a three-dimensional scene map.

[0006] In a first aspect, the embodiments of the present application provide a method for image processing. The method includes: obtaining an image to be processed and text description information used to describe the image to be processed; performing an isometric projection transformation process on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, where the target semantic segmentation map is used to reflect the specific meaning of the markers in the image to be processed; performing a diffusion process on the text description information and the target semantic segmentation map based on a target diffusion generation model to obtain a target projection scene map, where the target projection scene map corresponds to the text description information and the image to be processed, and the target diffusion generation model is a machine learning model obtained by training an initial diffusion generation model with a sample projection scene map and a label projection scene map, the sample projection scene map is obtained by performing a diffusion process on sample description information and a sample semantic segmentation map by the initial diffusion generation model, and the sample semantic segmentation map is obtained by performing an isometric projection transformation process on a sample image described by the sample description information; generating a target three-dimensional scene map based on the target projection scene map and the markers.

[0007] In a second aspect, an embodiment of the present application provides a method for model training. The method for model training includes: obtaining sample description information, a sample semantic segmentation map, and a label projection scene map, where the sample semantic segmentation map is obtained by performing an isometric projection transformation on a sample image, and the sample description information is used to describe the sample image; performing diffusion processing on the sample description information and the sample semantic segmentation map based on an initial diffusion generation model to obtain a sample projection scene map; updating the model parameters of the initial diffusion generation model based on the sample projection scene map and the label projection scene map to obtain a target diffusion generation model, where the target diffusion generation model is used to perform diffusion processing on text description information and a target semantic segmentation map to obtain a target projection scene map, the target projection scene map corresponds to the text description information and the image to be processed, the target semantic segmentation map is obtained by performing an isometric projection transformation on the image to be processed, and the text description information is used to describe the image to be processed.

[0008] In a third aspect, an embodiment of the present application provides an image processing apparatus. The image processing apparatus includes an acquisition unit, a processing unit, and a generation unit. Among them, the acquisition unit is used to acquire an image to be processed and text description information for describing the image to be processed. The processing unit is used to perform an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, where the target semantic segmentation map is used to reflect the specific meaning of the identification object in the image to be processed. The processing unit is used to perform diffusion processing on the text description information and the target semantic segmentation map based on the target diffusion generation model to obtain a target projection scene map, where the target projection scene map corresponds to the text description information and the image to be processed, and the target diffusion generation model is a machine learning model obtained by training an initial diffusion generation model with the sample projection scene map and the label projection scene map as training samples. The sample projection scene map is obtained by performing diffusion processing on the sample description information and the sample semantic segmentation map by the initial diffusion generation model, and the sample semantic segmentation map is obtained by performing an isometric projection transformation on the sample image described by the sample description information. The generation unit is used to generate a target three-dimensional scene map based on the target projection scene map and the identification object.

[0009] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the obtaining unit is further configured to: before performing an isometric projection transformation process on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, obtain target perspective information, where the target perspective information is used to reflect the value of the perspective parameter of the camera projection model under the isometric projection perspective. The processing unit is configured to: determine a target homography matrix based on the image to be processed and the target perspective information, where the target homography matrix is used to indicate the transformation situation when the image to be processed is transformed to the isometric projection perspective corresponding to the target perspective information; perform an isometric projection transformation process on the image to be processed based on the target homography matrix to obtain a target semantic segmentation map corresponding to the image to be processed.

[0010] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit is configured to: determine at least four first coordinate information, where each first coordinate information is used to indicate the pixel coordinate of a first vertex in the image coordinate system on the image to be processed; calculate the corresponding second coordinate information based on the target perspective information and each first coordinate information, where each second coordinate information is used to indicate the pixel coordinate of a second vertex in the image coordinate system, and each second vertex has a one-to-one correspondence with the corresponding first vertex; determine the target homography matrix based on all the first coordinate information and all the second coordinate information.

[0011] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit is configured to: perform a coordinate transformation process on each first coordinate information according to the value of the perspective parameter indicated by the target perspective information to calculate the corresponding second coordinate information.

[0012] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit is configured to: construct at least four groups of coordinate information pairs based on each first coordinate information and the corresponding second coordinate information; perform a singular value decomposition process on the at least four groups of coordinate information pairs to obtain the target homography matrix.

[0013] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit is configured to: perform a coordinate transformation process on the pixel coordinate of each pixel point in the image to be processed based on the target homography matrix to obtain the target pixel information of the corresponding pixel point; generate a target semantic segmentation map corresponding to the image to be processed based on the target pixel information of all the pixel points.

[0014] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the target diffusion generation model includes a ControlNet model, and the ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer; a processing unit, configured to: process the text description information and the target semantic segmentation map respectively based on the encoding layer to obtain a text feature vector and a semantic feature vector; process the text feature vector and the semantic feature vector based on the intermediate layer to obtain an intermediate layer feature vector; process the intermediate layer feature vector based on the convolution layer to obtain a convolution layer feature vector; use the zero convolution layer feature vector as the input of the output layer to obtain a target projection scene graph.

[0015] Fourthly, the embodiments of the present application provide a model training device. The model training device includes an acquisition module and a processing module. The acquisition module is configured to acquire sample description information, a sample semantic segmentation map, and a label projection scene graph, where the sample semantic segmentation map is obtained by performing an isometric projection transformation on a sample image, and the sample description information is used to describe the sample image. The processing module is configured to perform diffusion processing on the sample description information and the sample semantic segmentation map based on an initial diffusion generation model to obtain a sample projection scene graph. The processing module is configured to update the model parameters of the initial diffusion generation model based on the sample projection scene graph and the label projection scene graph to obtain a target diffusion generation model, where the target diffusion generation model is configured to perform diffusion processing on the text description information and the target semantic segmentation map to obtain a target projection scene graph, the target projection scene graph corresponds to the text description information and the image to be processed, the target semantic segmentation map is obtained by performing an isometric projection transformation on the image to be processed, and the text description information is used to describe the image to be processed.

[0016] In a possible design, in another implementation manner of the fourth aspect of the embodiments of the present application, the initial diffusion generation model includes a ControlNet model; the ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer; the processing module is specifically configured to: use the sample description information and the sample semantic segmentation map as the input of the encoding layer to obtain a sample text feature vector and a sample semantic feature vector; use the sample text feature vector and the sample semantic segmentation feature vector as the input of the intermediate layer to obtain an intermediate layer sample feature vector; process the intermediate layer sample feature vector based on the zero convolution layer to obtain a convolution layer sample feature vector; process the convolution layer sample feature vector based on the output layer to output a sample projection scene graph.

[0017] In a possible design, in another implementation manner of the fourth aspect of the embodiments of the present application, the processing module is configured to: calculate the image difference between the sample projection scene graph and the label projection scene graph to obtain a loss value; adjust the model parameters of the initial diffusion generation model based on the loss value to obtain a target diffusion generation model.

[0018] In a fifth aspect of the embodiments of the present application, a voice recognition device is provided, including: a memory, an input / output (I / O) interface, and a memory. The memory is used to store program instructions. The processor is used to execute the program instructions in the memory to execute the image processing method corresponding to the implementation manner of the first aspect above; or, execute the model training method corresponding to the implementation manner of the second aspect above.

[0019] In a sixth aspect of the embodiments of the present application, a computer-readable storage medium is provided. Instructions are stored in the computer-readable storage medium. When it runs on a computer, the computer is caused to execute the image processing method corresponding to the implementation manner of the first aspect above; or, execute the model training method corresponding to the implementation manner of the second aspect above.

[0020] In a seventh aspect of the embodiments of the present application, a computer program product containing instructions is provided. When it runs on a computer or a processor, the computer or the processor is caused to execute the image processing method corresponding to the implementation manner of the first aspect above; or, execute the model training method corresponding to the implementation manner of the second aspect above.

[0021] From the above technical solutions, it can be seen that the embodiments of the present application have the following advantages:

[0022] In the embodiments of the present application, after obtaining the image to be processed and the text description information for describing the image to be processed, first perform an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed. Through this target semantic segmentation map, the specific meaning of the identification object in the image to be processed can be reflected. In addition, the target diffusion generation model in the present application is a machine learning model obtained by training an initial diffusion generation model with a sample projection scene map and a label projection scene map as training samples. The mentioned sample projection scene map is obtained by performing diffusion processing on the sample description information and the sample semantic segmentation map by the initial diffusion generation model, and the sample semantic segmentation map is obtained by performing an isometric projection transformation on the sample image described by the sample description information. Then, after obtaining the target semantic segmentation map, use the target diffusion generation model to perform diffusion processing on the text description information and the target semantic segmentation map to obtain the corresponding target projection scene map. The target projection scene map corresponds to the text description information and the image to be processed. In this way, using the target projection scene map and the identification object, a target three-dimensional scene map can be generated.

[0023] In the above manner, after obtaining the text description information for describing the image to be processed in the embodiments of the present application, instead of directly performing image generation processing on the text description information using a diffusion model as in traditional image generation schemes, the isometric projection transformation processing of the image to be processed is first completed using the geometric relationship of isometric projection, not only obtaining the target semantic segmentation map including the specific meaning of the identifier, but also being able to fix the viewing point of the target semantic segmentation map under the isometric projection view angle. In this way, taking the target semantic segmentation map and the text description information as constraint conditions, and then using the target diffusion generation model to generate a target projection scene map that meets the constraint conditions and has a known viewing point, reducing the image deviation between the target projection scene map and the image actually described by the text description information, and facilitating the rapid and efficient construction of a three-dimensional scene map. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0025] Figure 1 Shows a schematic diagram of an image generated using a traditional image generation scheme;

[0026] Figure 2 Shows a schematic diagram of the system framework for image processing provided by the present application;

[0027] Figure 3 Shows a schematic diagram of an image application scenario provided by the present application;

[0028] Figure 4 Shows a schematic diagram of the flow of the method for model training provided by the present application;

[0029] Figure 5 Shows a schematic diagram of a sample image and a sample semantic segmentation map provided by the present application;

[0030] Figure 6 Shows a schematic diagram of the flow of the method for image processing provided by the present application;

[0031] Figure 7 Shows a schematic diagram of the image to be processed provided by the present application;

[0032] Figure 8 Shows a schematic diagram of the isometric projection transformation processing provided by the present application;

[0033] Figure 9 Shows a schematic diagram of the target projection scene map provided by the present application;

[0034] Figure 10 A schematic diagram showing an optional functional module structure of the image processing apparatus provided by the present application;

[0035] Figure 11 A schematic diagram showing an optional functional module structure of the model training apparatus provided by the present application;

[0036] Figure 12 A schematic diagram showing an optional hardware structure of the image processing device provided in the embodiments of the present application. Detailed implementation manners

[0037] Embodiments of the present application provide a method for image processing, a method for model training, and related apparatuses, which are used to generate a target projection scene graph that meets the conditions, so as to reduce image deviation and achieve fast and efficient construction of a three-dimensional scene graph.

[0038] It can be understood that in the specific implementation manners of the present application, data related to user information and the like are involved. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards of relevant countries and regions.

[0039] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0040] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above accompanying drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily need to be limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0041] With the research and progress of artificial intelligence (AI) technology, AI technology has been studied and applied in multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, driverless, autonomous driving, drones, robots, intelligent healthcare, intelligent customer service, etc. It is believed that with the development of technology, AI technology will be applied in more fields and play an increasingly important role.

[0042] Based on the development of AI technology, AI technology is also gradually used to complete image generation and processing in different scenarios. For example, in a virtual game scenario, AI technology can be used to generate a three-dimensional virtual game scene map; or, in an assisted driving scenario, AI technology can also be used to generate a navigation map during driving; or, in an intelligent transportation scenario, AI technology is used to generate road maps, road scene maps, etc.; or, in a home design scenario, AI technology can also be used to generate a home scene map that meets the user's needs. Specifically, the usage scenarios in the embodiments of this application are not limited.

[0043] However, in traditional image generation schemes, only the text description information used to describe the scene is used as the input of the diffusion model. For example, if the user wants a scene map with "grasslands on the periphery and a lake in the middle". At this time, only the text description information "grasslands on the periphery and a lake in the middle" is used as the input of the diffusion model, and it is easy to output an image with an unfixed viewpoint using this traditional diffusion model. And the unfixed viewpoint of the image means that for the same text description information, images that do not match the actual description are likely to appear.

[0044] For example, Figure 1 shows a schematic diagram of an image generated using a traditional image generation scheme. From Figure 1 the images shown in part (a) and part (b), it can be seen that although these two images are for the same text description information, their viewpoints are not fixed. For example, the image corresponding to part (a) is a scene map generated for a top-down view of 45°, while the image corresponding to part (b) is a scene map generated for a front view. This results in an inability to clearly determine the image actually described by the text description information and an inability to generate a three-dimensional scene map, thereby leading to a poor construction efficiency of the three-dimensional scene.

[0045] Therefore, to solve the above-mentioned technical problems, the embodiments of the present application provide a method for image processing. Correspondingly, the embodiments of the present application also provide a method for model training. After training a target diffusion generation model through the model training method provided by the present application, and then in the execution of the image processing method, by means of the target diffusion generation model, on the basis of considering the text description information of the image to be processed, further use the target semantic segmentation map corresponding to the image to be processed for constraint, so as to efficiently and quickly generate a target projection scene map with a fixed viewpoint, reduce the image deviation between the target projection scene map and the image actually described by the text description information, facilitate the rapid and efficient construction of a three-dimensional scene map, and meet the image generation requirements in different scenarios.

[0046] Exemplarily, the model training method and the image processing method provided by the embodiments of the present application are both implemented based on artificial intelligence. Artificial intelligence is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, a theory, method, technology, and application system that perceives the environment, acquires knowledge, and uses knowledge to obtain the best results. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technologies and software-level technologies. Artificial intelligence basic technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model, and can be widely applied to downstream tasks in major directions of artificial intelligence after fine-tuning. Artificial intelligence software technologies mainly include several major directions such as computer vision technology, speech technology, natural language processing technology, and machine learning / deep learning.

[0047] In the embodiments of the present application, the artificial intelligence technologies mainly involved include the above-mentioned machine learning, computer vision (CV) technology, natural language processing (NLP), etc. For example, it can involve text processing, semantic understanding, etc. in natural language processing; it can also involve deep learning in machine learning (ML), including autoencoders, artificial neural networks, etc., which are not specifically described in the embodiments of the present application. The embodiments of the present application also involve the above-mentioned computer vision technology, such as including but not limited to image processing and other technologies.

[0048] The method for model training provided in this application can be applied to an image processing device with data processing capabilities. Exemplarily, the method for image processing provided in this application can also be applied to the above-mentioned image processing device. As a schematic description, the mentioned image processing device includes but is not limited to terminal devices, servers, question-and-answer robots, etc. Among them, terminal devices can include but are not limited to smartphones, desktop computers, laptop computers, tablet computers, smart speakers, in-vehicle devices, smart watches, wearable smart devices, intelligent voice interaction devices, smart home appliances, aircraft, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, etc. A server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms, etc. This application does not make specific limitations. In addition, the mentioned terminal devices and servers can be directly or indirectly connected through wired communication or wireless communication, etc. This application does not make specific limitations.

[0049] The above-mentioned image processing device can be capable of implementing the above-mentioned natural language processing technology. The mentioned natural language processing technology is a kind of theory and method that studies how to effectively communicate between humans and computers in natural language. Natural language processing involves natural language, that is, the language used in daily life by people, and is closely related to linguistics research; at the same time, it involves computer science and mathematics. The pre-trained model, an important technology for model training in the field of artificial intelligence, is developed from the large language model (LLM) in the NLP field. After fine-tuning, the large language model can be widely applied to downstream tasks. Natural language processing technology generally includes technologies such as text processing, semantic understanding, machine translation, robot question and answer, and knowledge graph. In the embodiments of this application, the image processing device can implement processing such as semantic understanding on text description information through this natural language processing technology.

[0050] The above-mentioned image processing device may also be capable of implementing computer vision technology. Computer vision refers to machine vision that uses cameras and computers to replace human eyes for tasks such as object recognition, tracking, and measurement, and further performs image processing to make the computer-processed images more suitable for human eye observation or transmission to instruments for detection. Computer vision technology generally includes technologies such as image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc. In this application, image processing can be achieved by leveraging technologies such as image processing, video processing, three-dimensional object reconstruction, and 3D technology in computer vision technology.

[0051] The above-mentioned image processing device may also be capable of machine learning. Machine learning is an interdisciplinary field that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning generally include technologies such as neural networks. In the model training method and image processing method provided in the embodiments of this application, the use of artificial intelligence models mainly involves the application of neural networks. Through neural networks, the sample description information and sample semantic segmentation map are processed to obtain the sample projection scene map; further, through neural networks, the sample projection scene map and the label projection scene map are trained and processed to train the target diffusion generation model. In this way, the target diffusion generation model is then used to implement image processing on the text description information and target semantic segmentation map corresponding to the image to be processed.

[0052] Exemplarily, Figure 2 shows a schematic diagram of the system framework for image processing provided in this application. As Figure 2 shown, in this system framework, it includes a model training stage and a model usage stage.

[0053] Among them, in the model training stage, it is necessary to first obtain training samples. In these training samples, there are sample description information, sample semantic segmentation maps, and label projection scene maps. The mentioned sample description information is used to describe the sample images. In addition, the sample semantic segmentation maps are obtained by performing an isometric projection transformation on the sample images.

[0054] After obtaining the training samples, the sample description information and the sample semantic segmentation maps are used as the input of the initial diffusion generation model to perform diffusion processing through this initial diffusion generation model to obtain the sample projection scene maps. Further, the sample projection scene maps and the label projection scene maps are used to adjust and update the model parameters of the initial diffusion generation model to train the target diffusion generation model.

[0055] It should be noted that the described isometric projection transformation process can be understood as rotating a spatial object (referring to a sample image or an image to be processed in this application) around two coordinate axes contained in a certain projection plane, and then performing an isometric projection process onto this projection plane, thereby obtaining the isometric projection diagram of the spatial object. The mentioned isometric projection diagram can be understood as the orthographic axonometric drawing obtained when ηx = ηy = ηz. The mentioned ηx, ηy, and ηz are the axial deformation coefficients of the X-axis, the axial deformation coefficient of the Y-axis, and the axial deformation coefficient of the Z-axis, respectively. Exemplarily, the target projection scene diagram mentioned in this application can sometimes also be referred to as an orthographic axonometric projection scene diagram, and the specific name is not limited in this application.

[0056] In this way, during the model usage stage, the image to be processed and the corresponding text description information can be obtained first. Further, the orthographic axonometric projection transformation of the image to be processed is completed to obtain the corresponding target semantic segmentation diagram. The target diffusion generation model trained during the model training stage is used to process the target semantic segmentation diagram and the text description information to generate the corresponding target projection scene diagram.

[0057] In other words, during the model training stage, the sample description information and the sample semantic segmentation diagram corresponding to the sample image are used to jointly constrain the model parameters of the initial diffusion generation model, thereby realizing the fixation of the viewpoints of different images under the same perspective. In this way, subsequently, the trained target diffusion generation model can generate a target projection scene diagram that conforms to the constraints of the target semantic segmentation diagram and the text description information and has a fixed viewpoint, greatly reducing the image deviation and enabling the rapid and efficient construction of a three-dimensional scene diagram.

[0058] Figure 3 shows a schematic diagram of the image application scenario provided by this application. As Figure 3 shown, this application scenario at least includes an image processing device and a database. The image processing device can access the database through a network; alternatively, the database can also be integrated in the image processing device. Among them, the database is mainly used to store training samples, including but not limited to sample description information, sample projection scene diagrams, labeled projection scene diagrams, sample semantic segmentation diagrams, etc.; alternatively, it can also store the image to be processed and the text description information corresponding to the image to be processed, etc. The image processing device can obtain the image to be processed and the text description information of the image to be processed by accessing this database. In this way, by means of the target diffusion generation model trained in the foregoing Figure 2 , the text description information and the target semantic segmentation diagram corresponding to the image to be processed are processed, thereby obtaining the target projection scene diagram corresponding to the image to be processed.

[0059] It should be noted that the above Figure 3For illustration purposes only, an example is given where the image to be processed and the text description information corresponding to the image to be processed are stored in a database. In practical applications, the text description information corresponding to the image to be processed can also be text description information generated in real time based on the image to be processed, and specific limitations are not described in the embodiments of the present application.

[0060] Exemplarily, the model training method and the image processing method provided in the present application can also be applied to scenarios such as cloud technology, artificial intelligence, intelligent transportation, assisted driving, big data, etc. Specific limitations are not made in the present application.

[0061] For example, the model training method and the image processing method provided in the embodiments of the present application can also be applied in a virtual game scenario. In a virtual game scenario, the method provided in the present application can be used to generate a projected game topographic map from the same perspective, and more specifically, an isometric projection game map can be generated; a corresponding three-dimensional game scene map can also be constructed by combining identifiers such as props and buildings in the virtual game scene. As a schematic description, the model training method and the image processing method provided in the present application can also be applied to the user-generated content (UGC) scenario in virtual games. In the UGC scenario, users can execute the solution of the present application through a terminal device, etc., and can quickly grasp the game needs of users, etc., and can quickly and efficiently build a virtual scene, creating a UGC scene map that shows the user's needs, such as the in-game UGC scene map, etc., and no limitations are made here. The mentioned in-game UGC can be understood as users using the built-in DIY functions or tools in virtual games to create content.

[0062] Or, in an intelligent transportation scenario, the method provided in the present application can also be used to generate a projected navigation map, a projected navigation road sign map, etc. from the same perspective, and a corresponding three-dimensional navigation scene map can also be constructed by combining identifiers such as road signs and buildings on the corresponding road. Specific limitations on the usage scenario are not described in the embodiments of the present application.

[0063] As a schematic description, since the execution and processing of the above-described image processing method need to rely on the target diffusion generation model trained by the model training method in the early stage. Therefore, below, from the perspective of embodiments, the model training method provided in the embodiments of the present application will be described in detail. Exemplarily, Figure 4 shows a schematic flowchart of the model training method provided in the present application. As Figure 4 shown, the model training method at least includes the following steps:

[0064] 401. Obtain sample description information, a sample semantic segmentation map, and a label projection scene map. The sample semantic segmentation map is obtained by performing an isometric projection transformation on the sample image, and the sample description information is used to describe the sample image.

[0065] In this example, the sample image can be understood as an image drawn by an object such as a designer or an image user for the sample description information. For example, it includes but is not limited to hand-drawn images, or images made by drawing software, etc., which are not limited here. Exemplarily, the mentioned sample image can be an image from a top-down perspective, or an image from a bottom-up perspective; or an image from a front view, etc., which is not limited in the specific embodiments of this application.

[0066] Taking the sample image from a top-down perspective as an example, a panoramic image can be collected over a large range through the top-down perspective, and more detailed image information can be obtained, laying a data foundation for determining the semantic segmentation map later.

[0067] The described sample description information can be used to describe the sample image from a text perspective. For example, the sample description information can include but is not limited to "grassland around, lake in the middle", "an isometric view of a small blue house sitting on top of a lush green hill, simulation of water splashes, river in a forest", etc., which is not limited in the embodiments of this application.

[0068] After the image processing device obtains the sample image and the corresponding sample description information, it also performs an isometric projection transformation on the sample image to determine the sample semantic segmentation map corresponding to the sample image. Through this sample semantic segmentation map, the specific meanings of each identifier in the sample image can be known. The specific meanings of the described identifiers can include the specific semantics and position information of the identifiers, etc. For example, taking the identifier in the sample image as a tree, its specific meanings can include the semantics such as the name of the tree and its specific position in the sample image.

[0069] As a schematic description, in the process of calculating the sample semantic segmentation map corresponding to the sample image, since data such as topographic maps and semantic segmentation maps are missing in virtual game scenes, for example, in this application, procedural generation technology can be used to generate a large number of random three-dimensional scenes by 3D software. Through procedural generation technology, not only can a three-dimensional model of the scene be obtained, but also depth maps, semantic scene maps, etc. from arbitrary perspectives can be obtained. In this way, based on the obtained three-dimensional model of the scene, the image processing device can use three-dimensional model processing tools such as Blender to preset the camera projection model and adjust the camera position to conform to the orthographic projection relationship in advance, so as to construct the target perspective information. Through this target perspective information, the perspective parameter values of the camera projection model under the orthographic projection perspective can be reflected.

[0070] In most scenarios, images such as the scene maps of virtual game scenes are on the same plane. Therefore, in the embodiments of this application, the homography matrix is used to more intuitively and easily convert the drawn sample image or the image to be processed into the corresponding semantic segmentation map under the target perspective. It should be noted that the target perspective mentioned in this application is sometimes also called the orthographic projection perspective, and no name limitation is made specifically in this application.

[0071] Based on this, the image processing device determines the sample homography matrix based on the sample image and the target perspective information, and then performs an orthographic projection transformation process on the sample image based on this sample homography matrix to obtain the target semantic segmentation map corresponding to the sample image. For example, based on this sample homography matrix, coordinate transformation processing can be performed on the pixel coordinates of each pixel point in the sample image to obtain the target pixel information of the corresponding pixel point. After obtaining the target pixel information of all pixel points in the sample image, based on the target pixel information of all these pixel points, the sample semantic segmentation map corresponding to the sample image is generated.

[0072] For example, Figure 5 shows a schematic diagram of the sample image and the sample semantic segmentation map provided by this application. As Figure 5 shown, taking a virtual game scene as an example, if the sample description information is "an isometric view of a small blue house sitting on top of a lush green hill, simulation of water splashes, river in a forest". At this time, the sample image drawn by the object is as shown in part (a) of Figure 5 . After performing an orthographic projection transformation process on this sample image, its corresponding sample semantic segmentation map is as shown in Figure 5as shown in part (b) below. Starting from Figure 5 From the sample semantic segmentation map shown in part (b) below, it can be seen that it includes clearly seeing the specific positions, semantics, and other specific meanings of each identifier. For example, rivers and water in the forest, as well as small blue houses and trees located on the hillside, etc.

[0073] It should be noted that the described sample homography matrix can be used to indicate the transformation situation when the sample image is transformed to the isometric projection view corresponding to the target perspective information. In addition, for the process of solving the described sample homography matrix, it can be understood by referring to the solution process of the target homography matrix shown in the following Figure 8 and will not be elaborated here for the time being.

[0074] In addition, the above-mentioned label projection scene map can be understood as the real projection scene map corresponding to the sample description information.

[0075] 402. Perform diffusion processing on the sample description information and the sample semantic segmentation map based on the initial diffusion generation model to obtain a sample projection scene map.

[0076] In this example, after obtaining the training sample, the sample description information and the sample semantic segmentation map in the training sample can be used as the input of the initial diffusion generation model to complete the diffusion processing of the sample description information and the sample semantic segmentation map through the initial diffusion generation model, so as to obtain a sample projection scene map.

[0077] The described initial diffusion generation model may include, but is not limited to, the ControlNet model, etc., and is not limited in this application. As a schematic description, taking the ControlNet model as the initial diffusion generation model as an example, describe how to obtain a sample projection scene map. The described ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer.

[0078] The specific processing is as follows:

[0079] After obtaining the sample description information and the sample semantic segmentation map, first use the sample description information and the sample semantic segmentation map as the input of the encoding layer to perform vector encoding processing on the sample description information and the sample semantic segmentation map through the encoding layer, so as to obtain a sample text feature vector and a sample semantic feature vector. Through the sample semantic feature vector, the semantic situation of each identifier in the sample semantic segmentation map can be known.

[0080] After obtaining the sample text feature vector and the sample semantic feature vector, then use the sample text feature vector and the sample semantic feature vector as the input of the intermediate layer to complete information understanding and other processing of the sample text feature vector and the sample semantic feature vector through the intermediate layer, so as to obtain an intermediate layer sample feature vector.

[0081] Then, the intermediate layer sample feature vector is input into the zero convolution layer, and the zero convolution layer is used to process the intermediate layer sample feature vector to output the convolution layer sample feature vector. Finally, the output layer processes the convolution layer sample feature vector to output the sample projection scene graph. Exemplarily, in practical applications, sometimes the sample projection scene graph can also be referred to as the sample isometric projection topographic map, and the specific name is not limited.

[0082] By using the aforementioned ControlNet model to generate the sample projection scene graph, the overall and local composition of the sample projection scene graph can be controlled.

[0083] It should be noted that in the embodiments of the present application, only the ControlNet model is taken as an example of the initial diffusion generation model to illustrate how to obtain the sample projection scene graph. In practical applications, the initial diffusion generation model can also be other neural networks, which are not specifically limited in the present application.

[0084] 403. Based on the sample projection scene graph and the label projection scene graph, update the model parameters of the initial diffusion generation model to obtain a target diffusion generation model. The target diffusion generation model is used to perform diffusion processing on the text description information and the target semantic segmentation graph to obtain a target projection scene graph. The target projection scene graph corresponds to the text description information and the image to be processed. The target semantic segmentation graph is obtained by performing an isometric projection transformation on the image to be processed, and the text description information is used to describe the image to be processed.

[0085] In this example, after obtaining the sample projection scene graph, the sample projection scene graph and the label projection scene graph can be used as the input of the initial diffusion generation model to train the initial diffusion generation model, so as to obtain the target diffusion generation model.

[0086] As a schematic description, since it is desired that the output of the deep neural network is as close as possible to the value that is truly desired to be predicted, the weight vectors of each layer of the neural network can be updated by comparing the predicted value of the current network with the truly desired target value and then according to the difference between the two (of course, there is usually an initialization process before the first update, that is, parameters are preconfigured for each layer in the deep neural network). For example, if the predicted value of the network is high, the weight vector is adjusted to make it predict lower, and continuous adjustment is made until the neural network can predict the truly desired target value. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function. They are important equations for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference. Then the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0087] Therefore, during the specific training process, the loss function can be added synchronously to improve the learning ability of the diffusion generation model. In other words, after determining the sample projection scene graph, the image difference between the sample projection scene graph and the label projection scene graph is calculated to obtain a loss value. In this way, after obtaining the loss value, the model parameters of the initial diffusion generation model are adjusted based on the loss value to obtain the target diffusion generation model.

[0088] For example, the cross-entropy loss function can be used to calculate the image difference between the sample projection scene graph and the label projection scene graph to obtain the corresponding cross-entropy loss value. In addition, in addition to using the cross-entropy loss function to calculate the loss value mentioned above, other loss functions can also be used in practical applications to calculate the image difference between the sample projection scene graph and the label projection scene graph, which is not specifically limited in the embodiments of the present application.

[0089] After updating to obtain the target diffusion generation model, the target diffusion generation model can be used to process the text description information and the target semantic segmentation graph of the image to be processed to determine a qualified target projection scene graph, so as to generate a corresponding target three-dimensional scene graph using the target projection scene graph and the identification object.

[0090] The above Figure 4 mainly describes the model training method provided by the embodiments of the present application. After training the target diffusion generation model based on Figure 4 the described method, in the method of performing image processing, the target projection scene graph can be determined with the help of the target diffusion generation model. Exemplarily, Figure 6 shows a schematic flowchart of the image processing method provided by the present application.

[0091] As Figure 6 shown, the method for image processing provided by this application at least includes the following steps:

[0092] 601. Obtain the image to be processed and the text description information for describing the image to be processed.

[0093] In this example, the image to be processed can be understood as an image drawn by an object such as a designer or an image user for the text description information. For example, it includes but is not limited to hand-drawn images or images made by drawing software, etc., which are not limited here. Exemplarily, the mentioned image to be processed can be an image from a top view, or an image from a bottom view; or an image from a front view, etc., which are not limited in the specific embodiments of this application. The described text description information can be used to describe the image to be processed from a text perspective.

[0094] For example, Figure 7 shows a schematic diagram of the image to be processed provided by this application. As Figure 7 shown, taking a virtual game scene as an example, assuming the text description information is "There is grassland around and a lake in the middle", at this time, the object can use a hand-drawing tool or other drawing tools to simply draw the game scene image to be processed corresponding to the text description information.

[0095] 602. Perform an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, and the target semantic segmentation map is used to reflect the specific meaning of the identification objects in the image to be processed.

[0096] In this example, after obtaining the image to be processed, it is also necessary to perform an isometric projection transformation on the image to be processed, and then determine the target semantic segmentation map corresponding to the image to be processed. It should be noted that through the target semantic segmentation map, the specific meaning of the identification objects in the image to be processed can be reflected. For example, Figure 7 the identification object corresponding to part A in

[0097] is a lake, and the identification objects corresponding to the remaining parts are grasslands. Figure 2 From the content described above, the described isometric projection transformation is understood as rotating a spatial object around two coordinate axes contained in a certain projection plane and then performing an isometric projection on the projection plane to obtain the isometric projection map of the spatial object. And combining the above Figure 4As can be seen from the content described in step 401, images in most scenarios are on the same plane. Therefore, during the process of performing the orthographic projection transformation, the homography matrix can be used to achieve it. Specifically, regarding how to determine the target semantic segmentation map corresponding to the image to be processed, it can be understood with reference to the content described in the following steps S1 to S3, that is:

[0098] Step S1: Obtain the target view information, which is used to reflect the value of the view parameter of the camera projection model in the orthographic projection view.

[0099] In this example, the image processing device can, through a 3D model processing tool such as Blender, set the camera projection model in advance and adjust the camera position to conform to the orthographic projection relationship, so as to construct the target view information. For example, a view of 45° from above can be set in advance, etc., which is not limited here. Thus, in determining the target semantic segmentation map corresponding to the image to be processed, this target view information can be obtained first.

[0100] Step S2: Based on the image to be processed and the target view information, determine the target homography matrix, which is used to indicate the transformation situation when the image to be processed is transformed to the orthographic projection view corresponding to the target view information.

[0101] In this example, after obtaining the target view information, the image processing device can determine the target homography matrix based on the image to be processed and the target view information. Exemplarily, the image processing device can obtain at least four first vertices in the image to be processed from the image to be processed. Each first vertex does not coincide. In this way, after determining at least four first vertices, the image processing device can obtain the pixel coordinates of each first vertex in the image coordinate system to obtain the corresponding first coordinate information. In other words, the image processing device can determine at least four first coordinate information, and use each first coordinate information to indicate the pixel coordinates of a first vertex in the image to be processed in the image coordinate system.

[0102] After determining at least four first coordinate information, the image processing device then calculates the corresponding second coordinate information based on the target view information and each first coordinate information. As a schematic description, the image processing device performs coordinate transformation processing on each first coordinate information according to the value of the view parameter indicated by the target view information to calculate the corresponding second coordinate information. It should be noted that each second coordinate information can be used to indicate the pixel coordinates of a second vertex in the image coordinate system. In addition, each second vertex has a one-to-one correspondence with a first vertex.

[0103] In this way, after calculating the second coordinate information of the second vertex corresponding to each first vertex, the image processing device calculates the target homography matrix based on all the first coordinate information and all the second coordinate information. For example, the image processing device can form corresponding coordinate information pairs by combining each first coordinate information and the corresponding second coordinate information, so as to construct at least four groups of coordinate information pairs. After obtaining these at least four groups of coordinate information pairs, singular value decomposition (SVD) processing is performed on all these coordinate information pairs to obtain the target homography matrix.

[0104] For example, Figure 8 shows a schematic diagram of the isometric projection transformation process provided in this application. As Figure 8 shown, for the sake of simplicity in describing the process of calculating the target homography matrix, taking four vertices as an example, assume that the first coordinate information of the four first vertices (such as x 11 to x 41 ) in the image coordinate system of the image to be processed are respectively: x 11 (0, 0), x 21 (w, 0), x 31 (0, h), x 41 (w, h). Using the target viewing angle information to perform coordinate transformation on these four first vertices, the second coordinate information of the respective corresponding second vertices x 11 , x 21 , x 31 , x 41 in the image coordinate system can be determined as: x' 11 (0, h - w / 2 × tan30), x' 21 (w / 2, h - w × tan30), x' 31 (w / 2, h), x' 41 (w, h - w / 2 × tan30). Among them, x 11 corresponds to x' 11 , x 21 corresponds to x' 21 , x 31 corresponds to x' 31 , x 41 corresponds to x' 41 .

[0105] By defining the matrix B = H × A, where H represents the target homography matrix. After obtaining x 11 (0, 0), x 21 (w, 0), x 31 (0, h), x 41 (w, h), and x' 11(0, h - w / 2 × tan30), x' 21 (w / 2, h - w × tan30), x' 31 (w / 2, h), x' 41 After (w, h - w / 2 × tan30), input these four groups of coordinate information pairs into the above formula B = H × A. Specifically:

[0106]

[0107] Subsequently, by using the method of singular value decomposition, the matrix H can be calculated.

[0108] Step S3: Perform an isometric projection transformation on the image to be processed based on the target homography matrix to obtain the target semantic segmentation map corresponding to the image to be processed.

[0109] In this example, after calculating the target homography matrix, perform an isometric projection transformation on the image to be processed based on the target homography matrix to obtain the target semantic segmentation map corresponding to the image to be processed. Exemplarily, based on the target homography matrix, perform coordinate transformation on the pixel coordinates of each pixel point in the image to be processed to obtain the target pixel information of the corresponding pixel point. For example, the target pixel information of the pixel point satisfies the following formula where H is the target homography matrix, the (x, y) in represents the pixel information of the pixel point in the image to be processed, the (x'', y'') in represents the target pixel information of the pixel point. W represents the scaling factor, whose value is 1, is the homogeneous coordinate of the coordinate point in the transformed image, and W'' is the new scaling factor. In this way, after determining the target pixel information of all pixel points, based on the target pixel information of all pixel points, generate the target semantic segmentation map corresponding to the image to be processed, such as Figure 7 the target semantic segmentation map shown in.

[0110] 603. Perform diffusion processing on the text description information and the target semantic segmentation map based on the target diffusion generation model to obtain the target projection scene map. The target projection scene map corresponds to the text description information and the image to be processed. Among them, the target diffusion generation model is a machine learning model obtained by training the initial diffusion generation model with the sample projection scene map and the label projection scene map as training samples. The sample projection scene map is obtained by performing diffusion processing on the sample description information and the sample semantic segmentation map by the initial diffusion generation model, and the sample semantic segmentation map is obtained by performing an isometric projection transformation on the sample image described by the sample description information.

[0111] In this example, the described target diffusion generation model is a machine learning model obtained by training an initial diffusion generation model using a sample projection scene graph and a label projection scene graph as training samples. The specific training process can be understood by referring to the Figure 4 method for model training described above, which will not be elaborated here. Additionally, for the content such as the sample projection scene graph, label projection scene graph, sample description information, sample semantic segmentation graph, sample image, etc. mentioned here, reference can be made to the Figure 4 content described in steps 401 to 403 above, which will not be elaborated here.

[0112] After determining the target semantic segmentation graph, the text description information and the target semantic segmentation graph are used as inputs to the target diffusion generation model, so as to perform diffusion processing on the text description information and the target semantic segmentation graph through the target diffusion generation model to obtain a target projection scene graph.

[0113] Exemplarily, the described target diffusion generation model may include, but is not limited to, the ControlNet model, etc., which is not limited in this application. As a schematic description, taking the ControlNet model as the target diffusion generation model as an example, how to obtain the target projection scene graph is described. The described ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer. The specific processing is as follows:

[0114] After obtaining the text description information and the target semantic segmentation graph, first use the text description information and the target semantic segmentation graph as inputs to the encoding layer, so as to perform vector encoding processing on the text description information and the target semantic segmentation graph through the encoding layer to obtain a text feature vector and a semantic feature vector. Through the semantic feature vector, the semantic situation of each identifier in the target semantic segmentation graph can be known.

[0115] After obtaining the text feature vector and the semantic feature vector, then use the text feature vector and the semantic feature vector as inputs to the intermediate layer, so as to complete processing such as information understanding of the text feature vector and the semantic feature vector through the intermediate layer to obtain an intermediate layer feature vector.

[0116] In this way, then input the intermediate layer feature vector into the zero convolution layer, and further use the zero convolution layer to process the intermediate layer feature vector to output a convolution layer feature vector. Finally, the output layer processes the convolution layer feature vector to output the target projection scene graph. For example, Figure 9 shows a schematic diagram of the target projection scene graph provided by this application. As Figure 9 shown, for the Figure 7 pending image shown above, its corresponding isometric projection game topographic map can be generated. For example, the isometric projection game topographic map may include grassland and a lake in the middle of the grassland.

[0117] Exemplarily, in practical applications, sometimes the target projection scene graph can also be referred to as the target isometric projection topographic map, and there is no specific name limitation. By using the above-mentioned ControlNet model to generate the target projection scene graph, the overall and local composition of the target projection scene graph can be controlled, so as to control the generation of pictures that meet the needs of users.

[0118] It should be noted that in the embodiments of the present application, only the ControlNet model is taken as an example of the target diffusion generation model to illustrate how to obtain the sample projection scene graph. In practical applications, the target diffusion generation model can also be other neural networks, which are not specifically limited in the present application.

[0119] 604. Generate a target three-dimensional scene graph based on the target projection scene graph and the identifier.

[0120] In this example, since the three-dimensional scene graph includes not only the scene graph but also identifiers such as buildings, roads, objects, and props included in the scene. Therefore, in the process of generating the target three-dimensional scene graph, it is also necessary to extract the identifiers in the to-be-processed image, and then, after generating the target projection scene graph, generate the target three-dimensional scene graph based on the target projection scene graph and the identifiers. For example, three-dimensional scene construction tools such as 3D modeling software (including but not limited to) can be used to construct a three-dimensional scene for the target projection scene graph and the identifiers to generate the target three-dimensional scene graph. Through the target three-dimensional scene graph, the real scene can be clearly simulated, enabling the object to more intuitively understand the corresponding scene information, with strong interactivity and appreciation, and it is easy to meet the scene requirements of different objects.

[0121] In the embodiments of the present application, instead of directly using a diffusion model to perform image generation processing on text description information as in traditional image generation schemes, the isometric projection transformation processing of the to-be-processed image is first completed using the geometric relationship of isometric projection, not only obtaining the target semantic segmentation graph including the specific meaning of the identifier, but also being able to fix the viewing point of the target semantic segmentation graph under the isometric projection view angle. In this way, taking the target semantic segmentation graph and the text description information as constraint conditions, and then using the target diffusion generation model to generate a target projection scene graph that meets the constraint conditions and has a known viewing point, reducing the image deviation between the target projection scene graph and the image actually described by the text description information, and facilitating the rapid and efficient construction of the three-dimensional scene graph.

[0122] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of methods. It can be understood that in order to implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the modules and algorithm steps of each example described in the embodiments disclosed in the present application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the way of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0123] The embodiments of the present application can divide the device into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there can be other division methods in actual implementation.

[0124] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be fully or partially implemented by using software, hardware (such as a processing circuit or a memory), or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0125] The following will describe in detail the image processing device in the embodiments of the present application. Figure 10 It is a schematic diagram of the optional functional module structure of the image processing device provided in the embodiments of the present application. As Figure 10 shown, the image processing device may include an acquisition unit 1001, a processing unit 1002, and a generation unit 1003.

[0126] Among them, the acquisition unit 1001 is used to acquire the image to be processed and the text description information for describing the image to be processed. Specifically, it can be understood by referring to the content described in step 601 above, and details are not described here. Figure 6 For details, please refer to the content described in step 601 above and will not be elaborated here.

[0127] The processing unit 1002 is used to perform an isometric projection transformation process on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, and the target semantic segmentation map is used to reflect the specific meaning of the identification object in the image to be processed. Specifically, it can be referred to the aboveFigure 6 Understand the content described in step 602 in the foregoing, which will not be elaborated here.

[0128] The processing unit 1002 is configured to perform diffusion processing on the text description information and the target semantic segmentation map based on the target diffusion generation model to obtain a target projected scene map. The target projected scene map corresponds to the text description information and the image to be processed. The target diffusion generation model is a machine learning model obtained by training an initial diffusion generation model with a sample projected scene map and a labeled projected scene map as training samples. The sample projected scene map is obtained by performing diffusion processing on the sample description information and the sample semantic segmentation map by the initial diffusion generation model. The sample semantic segmentation map is obtained by performing an isometric projection transformation on the sample image described by the sample description information. Specifically, reference can be made to the content described in step 603 in the foregoing Figure 6 Understand the content described in step 603 in the foregoing, which will not be elaborated here.

[0129] The generation unit 1003 is configured to generate a target three-dimensional scene map based on the target projected scene map and the marker. Specifically, reference can be made to the content described in step 604 in the foregoing Figure 6 Understand the content described in step 604 in the foregoing, which will not be elaborated here.

[0130] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the acquisition unit 1001 is further configured to: before performing an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, acquire target view information, where the target view information is used to reflect the view parameter values of the camera projection model at the isometric projection view angle. The processing unit 1002 is configured to: determine a target homography matrix based on the image to be processed and the target view information, where the target homography matrix is used to indicate the transformation situation when the image to be processed is transformed to the isometric projection view angle corresponding to the target view information; perform an isometric projection transformation on the image to be processed based on the target homography matrix to obtain a target semantic segmentation map corresponding to the image to be processed.

[0131] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit 1002 is configured to: determine at least four first coordinate information, where each first coordinate information is used to indicate the pixel coordinates of a first vertex in the image coordinate system on the image to be processed; calculate corresponding second coordinate information based on the target view information and each first coordinate information, where each second coordinate information is used to indicate the pixel coordinates of a second vertex in the image coordinate system, and each second vertex has a one-to-one correspondence with the corresponding first vertex; determine the target homography matrix based on all the first coordinate information and all the second coordinate information.

[0132] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit 1002 is configured to: perform coordinate transformation processing on each first coordinate information according to the perspective parameter values indicated by the target perspective information, so as to calculate the corresponding second coordinate information.

[0133] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit 1002 is configured to: based on each first coordinate information and the corresponding second coordinate information, construct at least four groups of coordinate information pairs; perform singular value decomposition processing on the at least four groups of coordinate information pairs to obtain the target homography matrix.

[0134] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the processing unit 1002 is configured to: based on the target homography matrix, perform coordinate transformation processing on the pixel coordinates of each pixel point in the image to be processed, so as to obtain the target pixel information of the corresponding pixel point; generate a target semantic segmentation map corresponding to the image to be processed based on the target pixel information of all pixel points.

[0135] In a possible design, in another implementation manner of the third aspect of the embodiments of the present application, the target diffusion generation model includes a ControlNet model, and the ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer; the processing unit 1002 is configured to: respectively process the text description information and the target semantic segmentation map based on the encoding layer to obtain a text feature vector and a semantic feature vector; process the text feature vector and the semantic feature vector based on the intermediate layer to obtain an intermediate layer feature vector; process the intermediate layer feature vector based on the convolution layer to obtain a convolution layer feature vector; use the zero convolution layer feature vector as the input of the output layer to obtain the target projection scene map.

[0136] The above mainly describes the image processing device from the perspective of functional modules. Next, the optional functional module structure of the model training device provided by the present application will be described from the perspective of functional modules. As Figure 11 shown, the model training device may include an acquisition module 1101 and a processing module 1102.

[0137] Among them, the acquisition module 1101 is configured to acquire sample description information, a sample semantic segmentation map, and a label projection scene map. The sample semantic segmentation map is obtained by performing an isometric projection transformation on a sample image, and the sample description information is used to describe the sample image. Specifically, reference may be made to the content described in step 401 above Figure 4 for understanding, and details are not described here.

[0138] The processing module 1102 is configured to perform diffusion processing on the sample description information and the sample semantic segmentation map based on the initial diffusion generation model to obtain a sample projected scene map. Specifically, it can be understood with reference to the content described in step 402 above, which will not be elaborated here. Figure 4 For the understanding of the content described in step 402 above, no further elaboration will be provided here.

[0139] The processing module 1102 is configured to update the model parameters of the initial diffusion generation model based on the sample projected scene map and the label projected scene map to obtain a target diffusion generation model. The target diffusion generation model is used to perform diffusion processing on the text description information and the target semantic segmentation map to obtain a target projected scene map. The target projected scene map corresponds to the text description information and the image to be processed. The target semantic segmentation map is obtained by performing an isometric projection transformation on the image to be processed. The text description information is used to describe the image to be processed. Specifically, it can be understood with reference to the content described in step 403 above, which will not be elaborated here. Figure 4 For the understanding of the content described in step 403 above, no further elaboration will be provided here.

[0140] In a possible design, in another implementation manner of the fourth aspect of the embodiments of the present application, the initial diffusion generation model includes a ControlNet model; the ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer; the processing module 1102 is specifically configured to: use the sample description information and the sample semantic segmentation map as the input of the encoding layer to obtain a sample text feature vector and a sample semantic feature vector; use the sample text feature vector and the sample semantic segmentation feature vector as the input of the intermediate layer to obtain an intermediate layer sample feature vector; process the intermediate layer sample feature vector based on the zero convolution layer to obtain a convolution layer sample feature vector; process the convolution layer sample feature vector based on the output layer to output a sample projected scene map.

[0141] In a possible design, in another implementation manner of the fourth aspect of the embodiments of the present application, the processing module 1102 is configured to: calculate the image difference between the sample projected scene map and the label projected scene map to obtain a loss value; adjust the model parameters of the initial diffusion generation model based on the loss value to obtain a target diffusion generation model.

[0142] The above describes the image processing device in the embodiments of the present application from the perspective of modular functional entities. The following describes the image processing device in the embodiments of the present application from the perspective of hardware processing. Figure 12 It is a schematic diagram of an optional hardware structure of the image processing device provided in the embodiments of the present application. This image processing device may vary greatly due to configuration or performance differences, for example, including but not limited to the Figure 10 image processing device mentioned above, or Figure 11 the model training device mentioned above, and so on.

[0143] Such asFigure 12 As shown, the image processing device 300 can vary greatly due to different configurations or performances, and may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 342 or data 344. Among them, the memory 332 and the storage media 330 can be transient storage or persistent storage. The programs stored in the storage media 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the image processing device. Further, the central processing unit 322 can be set to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the image processing device 300. Exemplarily, the central processing unit 322 is used to execute the application programs stored in the storage media 330, so as to implement the model training method or the image processing method provided in the above embodiments of the present application.

[0144] The image processing device 300 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0145] Exemplarily, Figure 12 the central processing unit 322 in can make the image processing device execute the method in the corresponding method embodiment by calling the computer-executable instructions stored in the memory 332. Figures 4 to 6

[0146] Specifically, Figure 10 the functions / implementation processes of the processing unit 1002 and the generating unit 1003 in, Figure 11 the processing module 1102 in can be realized by the central processing unit 322 in calling the computer-executable instructions stored in the memory 332. Figure 12 Figure 10 the obtaining unit 1001 in and Figure 11 the obtaining module 1101 in the functions / implementation processes of can be realized by the input / output interface 358 in. Figure 12

[0147] The steps performed by the image processing device in the above embodiments can be based on the structure of the image processing device shown in this Figure 12

[0148] ​​​​In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0149] In an embodiment of the present application, a computer program product is further provided, including a computer program or instruction. When the computer program or instruction is executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0150] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product.

[0151] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0152] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, indirect couplings or communication connections of devices or units, and can be in electrical, mechanical, or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment. In addition, the functional units in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0154] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0155] A computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, they generate, in whole or in part, the processes or functions according to the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, an SSD), etc.

[0156] The above embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of this application.

Claims

1. A method for image processing, characterized in that, Including: Obtaining an image to be processed and text description information for describing the image to be processed; Performing an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, where the target semantic segmentation map is used to reflect the specific meaning of the identification object in the image to be processed; Performing diffusion processing on the text description information and the target semantic segmentation map based on a target diffusion generation model to obtain a target projection scene map, where the target projection scene map corresponds to the text description information and the image to be processed. Among them, the target diffusion generation model is a machine learning model obtained by training an initial diffusion generation model with a sample projection scene map and a label projection scene map as training samples. The sample projection scene map is obtained by performing diffusion processing on sample description information and a sample semantic segmentation map by the initial diffusion generation model, and the sample semantic segmentation map is obtained by performing an isometric projection transformation on a sample image described by the sample description information; Generating a target three-dimensional scene map based on the target projection scene map and the identification object.

2. The method according to claim 1, characterized in that, Before performing an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, the method further includes: Obtaining target perspective information, where the target perspective information is used to reflect the perspective parameter values of the camera projection model under the isometric projection perspective; Determining a target homography matrix based on the image to be processed and the target perspective information, where the target homography matrix is used to indicate the transformation situation when the image to be processed is transformed to the isometric projection perspective corresponding to the target perspective information; Performing an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, including: Performing an isometric projection transformation on the image to be processed based on the target homography matrix to obtain a target semantic segmentation map corresponding to the image to be processed.

3. The method according to claim 2, wherein Determining a target homography matrix based on the image to be processed and the target perspective information, including: Determining at least four first coordinate information, where each first coordinate information is used to indicate the pixel coordinates of a first vertex in the image to be processed on the image coordinate system; Calculating corresponding second coordinate information based on the target perspective information and each first coordinate information, where each second coordinate information is used to indicate the pixel coordinates of a second vertex on the image coordinate system, and each second vertex has a one-to-one correspondence with the corresponding first vertex; Determining a target homography matrix based on all the first coordinate information and all the second coordinate information.

4. The method according to claim 3, wherein Calculating corresponding second coordinate information based on the target perspective information and each first coordinate information, including: Performing coordinate transformation processing on each first coordinate information according to the perspective parameter values indicated by the target perspective information to calculate the corresponding second coordinate information.

5. According to the method described in any one of claims 3 to 4, determining a target homography matrix based on all the first coordinate information and all the second coordinate information, including: Based on each of the first coordinate information and the corresponding second coordinate information, at least four groups of coordinate information pairs are constructed; Perform singular value decomposition on the at least four groups of coordinate information pairs to obtain a target homography matrix.

6. The method according to any one of claims 2 to 5, characterized in that Perform an isometric projection transformation on the image to be processed based on the target homography matrix to obtain a target semantic segmentation map corresponding to the image to be processed, including: Based on the target homography matrix, perform coordinate transformation on the pixel coordinates of each pixel point in the image to be processed to obtain target pixel information corresponding to the pixel point; Generate a target semantic segmentation map corresponding to the image to be processed based on the target pixel information of all the pixel points.

7. The method according to any one of claims 1 to 6, characterized in that, The target diffusion generation model includes a ControlNet model, and the ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer; Process the text description information and the target semantic segmentation map based on the target diffusion generation model to obtain a target projection scene map, including: Process the text description information and the target semantic segmentation map respectively based on the encoding layer to obtain a text feature vector and a semantic feature vector; Process the text feature vector and the semantic feature vector based on the intermediate layer to obtain an intermediate layer feature vector; Process the intermediate layer feature vector based on the convolution layer to obtain a convolution layer feature vector; Use the zero convolution layer feature vector as the input of the output layer to obtain a target projection scene map.

8. A method for model training, characterized in that, Including: Obtain sample description information, a sample semantic segmentation map, and a labeled projection scene map. The sample semantic segmentation map is obtained by performing an isometric projection transformation on a sample image, and the sample description information is used to describe the sample image; Perform diffusion processing on the sample description information and the sample semantic segmentation map based on an initial diffusion generation model to obtain a sample projection scene map; Update the model parameters of the initial diffusion generation model based on the sample projection scene map and the labeled projection scene map to obtain a target diffusion generation model. The target diffusion generation model is used to perform diffusion processing on text description information and a target semantic segmentation map to obtain a target projection scene map. The target projection scene map corresponds to the text description information and the image to be processed. The target semantic segmentation map is obtained by performing the isometric projection transformation on the image to be processed, and the text description information is used to describe the image to be processed.

9. The method according to claim 8, wherein The initial diffusion generation model includes a ControlNet model; the ControlNet model includes an encoding layer, an intermediate layer, a zero convolution layer, and an output layer. Performing diffusion processing on the sample description information and the sample semantic segmentation map based on the initial diffusion generation model to obtain a sample projection scene map includes: Use the sample description information and the sample semantic segmentation map as the input of the encoding layer to obtain a sample text feature vector and a sample semantic feature vector; Use the sample text feature vector and the sample semantic segmentation feature vector as the input of the intermediate layer to obtain an intermediate layer sample feature vector; Process the intermediate layer sample feature vectors based on the zero convolution layer to obtain convolution layer sample feature vectors; Process the convolution layer sample feature vectors based on the output layer to output a sample projection scene graph.

10. The method according to any one of claims 8 to 9, characterized in that, Update the model parameters of the initial diffusion generation model based on the sample projection scene graph and the label projection scene graph to obtain a target diffusion generation model, including: Calculate the image difference between the sample projection scene graph and the label projection scene graph to obtain a loss value; Adjust the model parameters of the initial diffusion generation model based on the loss value to obtain a target diffusion generation model.

11. An image processing apparatus, characterized in that, Include: An acquisition unit for acquiring an image to be processed and text description information for describing the image to be processed; The processing unit is configured to perform an isometric projection transformation on the image to be processed to obtain a target semantic segmentation map corresponding to the image to be processed, and the target semantic segmentation map is used to reflect the specific meaning of the identification objects in the image to be processed; The processing unit is configured to perform diffusion processing on the text description information and the target semantic segmentation map based on the target diffusion generation model to obtain a target projection scene graph, and the target projection scene graph corresponds to the text description information and the image to be processed. The target diffusion generation model is a machine learning model obtained by training the initial diffusion generation model with the sample projection scene graph and the label projection scene graph as training samples. The sample projection scene graph is obtained by performing diffusion processing on the sample description information and the sample semantic segmentation map by the initial diffusion generation model, and the sample semantic segmentation map is obtained by performing an isometric projection transformation on the sample image described by the sample description information; A generation unit for generating a target three-dimensional scene graph based on the target projection scene graph and the identification object.

12. A model training device, characterized in that, Include: An acquisition module for acquiring sample description information, a sample semantic segmentation map, and a label projection scene graph. The sample semantic segmentation map is obtained by performing an isometric projection transformation on a sample image, and the sample description information is used to describe the sample image; A processing module for performing diffusion processing on the sample description information and the sample semantic segmentation map based on the initial diffusion generation model to obtain a sample projection scene graph; The processing module is configured to update the model parameters of the initial diffusion generation model based on the sample projection scene graph and the label projection scene graph to obtain a target diffusion generation model. The target diffusion generation model is used to perform diffusion processing on the text description information and the target semantic segmentation map to obtain a target projection scene graph, and the target projection scene graph corresponds to the text description information and the image to be processed. The target semantic segmentation map is obtained by performing the isometric projection transformation on the image to be processed, and the text description information is used to describe the image to be processed.

13. An image processing apparatus, characterized in that, Include: An input / output interface, a processor, and a memory, and program instructions are stored in the memory; The processor is used to execute program instructions stored in the memory, and execute the image processing method described in any one of claims 1 to 7; or execute the model training method described in any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when run on a computer device, cause the computer device to execute the image processing method described in any one of claims 1 to 7; or execute the model training method described in any one of claims 8 to 10.

15. A computer program product, characterized in that, The computer program product includes instructions that, when run on a computer device, cause the computer device to execute the image processing method described in any one of claims 1 to 7; or execute the model training method described in any one of claims 8 to 10.

Images