Image processing method, computer readable storage medium and electronic device

Abstract

The application discloses an image processing method, a computer readable storage medium and an electronic device. The method is applied to the field of image segmentation, and includes: obtaining a to-be-recognized image; performing target detection on the surface of at least one entity object in the to-be-recognized image to obtain a surface image of the surface of the at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; and performing semantic segmentation on the to-be-recognized image based on the image features of the surface image to obtain a semantic segmentation result of the surface of the at least one entity object. The application solves the problem that a plane estimation algorithm in the related art cannot obtain semantic information of a plane, thereby reducing the accuracy of image segmentation.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method, a computer-readable storage medium, and an electronic device. Background Technology

[0002] The task of plane estimation algorithms is to perform 3D geometric structure analysis on images and extract planar regions and plane parameters from the images. Here, a plane can refer to the surface of an object contained in the image, such as a tabletop, wall, or ground. This task is crucial for processes such as scene understanding, scene reconstruction, and 3D fusion rendering, and plays a powerful AI-enabled role in industries such as interactive entertainment, autonomous driving, smart homes, and AR (Augmented Reality) tourism.

[0003] Current plane estimation algorithms employ a multi-task training scheme that combines instance segmentation and depth estimation. They can predict plane parameters, plane segmentation masks, and scene depth maps separately, but they cannot obtain the semantic information of the plane. This results in the inability to accurately segment the surfaces of different objects during image segmentation, leading to low image segmentation accuracy.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an image processing method, a computer-readable storage medium, and an electronic device to at least solve the technical problem in the related art that plane estimation algorithms cannot obtain the semantic information of the plane, resulting in low accuracy of image segmentation.

[0006] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: acquiring an image to be identified, wherein the image to be identified includes the surface of at least one entity object; performing target detection on the surface of the at least one entity object in the image to be identified to obtain a surface image of the surface of the at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; and performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of the at least one entity object; wherein the semantic segmentation result includes: the category of the at least one entity object, the region in which the surface of the at least one entity object is located in the image to be identified, the planar parameters of the surface of the at least one entity object, and whether each pixel in the image to be identified belongs to the surface of the at least one entity object.

[0007] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: responding to an input command applied to an operation interface, displaying an image to be identified on the operation interface, wherein the image to be identified contains the surface of at least one entity object; responding to an instance segmentation command applied to the operation interface, displaying a semantic segmentation result of the surface of at least one entity object on the operation interface, wherein the semantic segmentation result is obtained by semantically segmenting the image to be identified based on the image features of the surface image of the surface of at least one entity object, the semantic segmentation result including: the category of at least one entity object, the region in the image to be identified where the surface of at least one entity object is located, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object, the surface image is obtained by target detection of the surface of at least one entity object in the image to be identified, and the image features of the surface image are obtained by performing image feature extraction on the surface image.

[0008] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: displaying an image to be identified on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image to be identified contains the surface of at least one entity object; performing target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object, wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object; and driving the VR device or AR device to display the semantic segmentation result of the surface of at least one entity object.

[0009] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: acquiring an image to be identified by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the image to be identified, and the image to be identified containing the surface of at least one entity object; performing target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object, wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object; and outputting the semantic segmentation result of the surface of at least one entity object by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the semantic segmentation result of the surface of at least one entity object.

[0010] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein, when the program is running, the device where the computer-readable storage medium is located executes any of the above methods.

[0011] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the method of any one of the above when it runs.

[0012] In this embodiment, after obtaining an image of the surface containing at least one entity object, target detection is first performed on the surface of the at least one entity object in the image to be identified, resulting in a surface image of the surface of the at least one entity object. Then, image feature extraction is performed on the surface image to obtain image features of the surface image. Furthermore, semantic segmentation is performed on the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object, thus achieving the purpose of planar semantic segmentation and parameter estimation. It is easy to note that when processing the image to be identified, this embodiment can not only perform target detection on the surface of at least one entity object in the image to be identified, but also perform semantic segmentation on the image to be identified based on the image features of different surface images. This achieves the technical effect of simultaneously obtaining planar parameters, planar segmentation masks, and semantic information, as well as the technical effect of accurately segmenting the surfaces of different objects. This solves the technical problem in related technologies where planar estimation algorithms cannot obtain planar semantic information, resulting in low image segmentation accuracy. Furthermore, in 3D fusion rendering scenarios, the background image in the image to be identified can be quickly and accurately determined based on the semantic segmentation result, achieving low-cost intelligent recognition of background images and virtual objects, and reducing the cost of 3D fusion.

[0013] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0014] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0015] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an embodiment of the image processing method of this application;

[0016] Figure 2 This is a structural block diagram of a computing environment for an image processing method according to an embodiment of this application;

[0017] Figure 3 This is a flowchart of an image processing method according to Embodiment 1 of this application;

[0018] Figure 4 This is a schematic diagram of an optional interactive interface according to an embodiment of this application;

[0019] Figure 5 This is a schematic diagram illustrating an optional image processing method according to an embodiment of this application;

[0020] Figure 6 This is a schematic diagram illustrating another optional image processing method according to an embodiment of this application;

[0021] Figure 7 This is a schematic diagram of an optional image processing method according to an embodiment of this application;

[0022] Figure 8 This is a flowchart of an image processing method according to Embodiment 2 of this application;

[0023] Figure 9 This is a flowchart of an image processing method according to Embodiment 3 of this application;

[0024] Figure 10 This is a flowchart of an image processing method according to Embodiment 4 of this application;

[0025] Figure 11 This is a schematic diagram of an image processing apparatus according to an embodiment of this application;

[0026] Figure 12 This is a schematic diagram of an image processing apparatus according to an embodiment of this application;

[0027] Figure 13 This is a schematic diagram of an image processing apparatus according to an embodiment of this application;

[0028] Figure 14 This is a schematic diagram of an image processing apparatus according to an embodiment of this application;

[0029] Figure 15 This is a structural block diagram of an AR / VR device according to an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0033] Mask: An image of the same size as the original image, marking each pixel as belonging to the foreground or background.

[0034] Instance segmentation: Segmenting the target instance regions in an image, such as tables and chairs.

[0035] Encoding network: A neural network structure in the field of deep learning that extracts deep features from images.

[0036] Region Proposal Network (RPN): A neural network structure used in deep learning object detection to extract candidate boxes.

[0037] This application provides an image processing method that can achieve planar semantic segmentation while estimating planar parameters.

[0038] Example 1

[0039] According to an embodiment of this application, an image processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 1 This is a schematic diagram of the hardware environment of a virtual reality device according to an embodiment of the image processing method of this application. Figure 1As shown, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 via a network. The virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc. The terminal 104 is not limited to PCs, mobile phones, tablets, etc. The server 102 can be a server corresponding to a media file operator. The network includes, but is not limited to: wide area network, metropolitan area network, or local area network.

[0041] Optionally, the virtual reality device 104 in this embodiment includes a memory, a processor, and a transmission device. The memory stores an application program that can be used to perform: acquiring an image to be recognized; performing target detection on the surface of at least one entity object in the image to be recognized to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; and performing semantic segmentation on the image to be recognized based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object. This solves the technical problem in related technologies where plane estimation algorithms cannot obtain semantic information of the plane, resulting in low accuracy of image segmentation, and achieves the goal of obtaining semantic segmentation results of surface images.

[0042] The terminal in this embodiment can be used to display an image to be identified on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image to be identified contains the surface of at least one entity object; perform target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; perform image feature extraction on the surface image to obtain image features of the surface image; perform semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object, wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object; and drive the VR device or AR device to display the semantic segmentation result of the surface of at least one entity object.

[0043] Optionally, the eye-tracking HMD (Head-Mounted Display) and eye-tracking module in the virtual reality device 104 of this embodiment function the same as in the embodiments described above. That is, the screen in the HMD is used to display real-time images, and the eye-tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The terminal in this embodiment acquires the user's position and movement information in real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in virtual three-dimensional space, as well as the user's field of vision orientation in virtual three-dimensional space.

[0044] Figure 1 The hardware structure block diagram shown can serve not only as an exemplary block diagram of the aforementioned AR / VR device (or mobile device), but also as an exemplary block diagram of the aforementioned server. In one optional embodiment, Figure 2 The use of the above is illustrated in a block diagram. Figure 1 The AR / VR device (or mobile device) shown is an example of a computing node in computing environment 201. Figure 2 This is a structural block diagram of a computing environment for an image processing method according to an embodiment of this application, such as... Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the diagram). Each computing node contains local processing and memory resources, and end user 202 can remotely run applications or store data within computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3, and 220-4 within computing environment 201, representing services "A", "D", "E", and "H", respectively.

[0045] End user 202 can provide and access services through a web browser or other software application on a client. In some embodiments, the provisioning and / or requests of end user 202 can be provided to ingress gateway 230. Ingress gateway 230 may include a corresponding agent to handle the provisioning and / or requests for services (one or more services provided in computing environment 201).

[0046] The services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services may be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. VM-based virtualization can simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the machine is virtualized by a virtual machine, container-based virtualization can launch containers to virtualize an entire operating system (OS), allowing multiple workloads to run on a single OS instance.

[0047] In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, such as Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Each Pod can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls service-related network functions such as routing and load balancing. Other services can also be Pods similar to Pods.

[0048] During operation, executing a user request from end user 202 may require calling one or more services in computing environment 201, and executing one or more functions of one service may require calling one or more functions of another service. For example... Figure 2 As shown, service "A" 220-1 receives user requests from terminal user 202 from ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can request service "E" 220-3 to perform one or more functions.

[0049] The aforementioned computing environment can be a cloud computing environment, where resource allocation is managed by cloud services, allowing functionality development without needing to consider implementation, adjustment, or server scaling. This computing environment allows developers to execute event-responsive code without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically and independently scale, rather than scaling a single hardware device to handle potential loads.

[0050] Under the aforementioned operating environment, this application provides the following: Figure 3 The image processing method shown is illustrated. It should be noted that the image processing method in this embodiment can be derived from... Figure 1The virtual reality device or augmented reality device of the illustrated embodiment is used for execution. Figure 3 This is a flowchart of an image processing method according to Embodiment 1 of this application. Figure 3 As shown, the method may include the following steps:

[0051] Step S302: Obtain the image to be identified, wherein the image to be identified contains the surface of at least one entity object.

[0052] The image to be identified can be a three-channel image containing the surface of one or more entity objects. The entity objects can be items in the real environment, such as tables, chairs, refrigerators, walls, floors, lamps, tableware, quilts, bags, etc. The surface of the entity object can be the surface of the item, such as the tabletop, the back of the chair, the seat of the chair, the front of the refrigerator, the wall, the ground, etc.

[0053] In one optional embodiment, the user's AR device, VR device, or mobile terminal can use its built-in camera to capture images of the real environment to obtain the image to be recognized, which can then be processed by the AR device, VR device, or mobile terminal. In another optional embodiment, the user's AR device, VR device, or mobile terminal can provide the user with an interactive interface. Figure 4 This is a schematic diagram of an optional interactive interface according to an embodiment of this application. For example... Figure 4 As shown, users can take a picture of the real environment by clicking the "Shoot" button to obtain the image to be recognized, or users can drag the image to be recognized into the dashed box to obtain the image to be recognized. Then, by clicking the "Plane Instance Segmentation" button, AR devices, VR devices, or mobile terminals can send the image to be recognized to the cloud server for processing.

[0054] It should be noted that the aforementioned mobile terminals may include, but are not limited to: smartphones (including Android phones and iOS phones), laptops, tablets, PDAs, etc.

[0055] For example, taking a 3D fusion rendering scene as an example, a user can use the VR device's built-in camera to take a picture of their room, obtaining an image of the room. Since there is a table in the room, the image includes three planes: the tabletop, the walls, and the floor. To reduce the computational load on the VR device, it can upload the room image to a cloud server for processing.

[0056] Step S304: Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object.

[0057] The surface image mentioned above can be an image of a surface containing only one entity object, obtained after object detection.

[0058] In an optional embodiment, to more accurately perform semantic segmentation on the surfaces of entity objects in the image to be identified, object detection can be performed on the image to be identified to obtain candidate bounding boxes corresponding to the surface of each entity object, that is, at least one candidate bounding box is obtained. Here, the candidate bounding box is used to characterize the position of the surface of an entity object in the image to be identified. Then, the image to be identified is cropped based on at least one candidate bounding box to obtain a surface image of the surface of each entity object. In this embodiment, object detection algorithms provided in related technologies can be used to detect planes in the image to be identified. For example, the RPN network provided in related technologies can be used to extract candidate bounding boxes in the image to be identified, and further obtain a surface image of the surface of each entity object.

[0059] It should be noted that since candidate boxes are usually rectangular, while the surface of an entity object is not necessarily rectangular, the surface image contains not only the surface of the entity object, but also other content. However, the other content does not affect the semantic segmentation process.

[0060] For example, taking a 3D fusion rendering scenario as an example, after the cloud server receives the room image uploaded by the VR device, it can perform object detection on the room image to obtain three candidate boxes. By cropping the images within the three candidate boxes, it can obtain three surface images: a desktop image, a wall image, and a floor image.

[0061] Step S306: Perform image feature extraction on the surface image to obtain the image features of the surface image.

[0062] The image features of the surface image mentioned above can be the features of the surface of the entity object in the surface image, that is, the features of the candidate box corresponding to the surface image. These features include, but are not limited to, the color features, texture features, shape features and spatial relationship features of the image.

[0063] In one optional embodiment, features are first extracted from the entire image to be identified using a feature extraction network to obtain image features F. Then, the candidate boxes obtained from the target detection process described above are used to crop and warp the image features F, thereby obtaining the image features R of the surface image. i .

[0064] It should be noted that the feature extraction network mentioned above can be any network provided in related technologies. In this embodiment, a backbone network is used as an example for illustration, but it is not limited to this.

[0065] In another alternative embodiment, Figure 5 This is a schematic diagram illustrating an optional image processing method according to an embodiment of this application, such as... Figure 5 As shown, a three-channel image can be input into a deep residual network (DRN) to extract features from the entire image to be recognized, obtaining the image features F of the entire image to be recognized. Then, through two branches, global pooling predicts plane parameters, and pyramid pooling predicts segmentation masks and non-planar depth maps through convolution and a conditional probability distribution model (CRF), thus obtaining a piecewise planar depth map.

[0066] In yet another alternative embodiment, Figure 6 This is a schematic diagram illustrating another optional image processing method according to an embodiment of this application, such as... Figure 6 As shown, a multi-task training scheme combining plane instance segmentation and monocular depth estimation is adopted. The instance segmentation algorithm outputs bounding boxes (BBoxes), plane segmentation results (Masks), and plane normals. The depth estimation algorithm outputs plane offsets. Then, through convolution (Conv), sharing, and summation operations of the ConvAccu Module, the segmentation result and plane parameters (normal + offset) of each plane instance are finally obtained. This scheme uses video data for training and improves the accuracy of plane segmentation and parameter estimation by calculating the 3D reconstruction loss and segmentation warping loss of adjacent video frames.

[0067] For example, taking a 3D fusion rendering scene as an example, the cloud server can extract features from the room image to obtain the image features of the entire room image. Then, based on the positions of the three candidate boxes corresponding to the three surfaces in the room image, the image features of the entire room image are cropped and deformed to obtain the desktop features of the desktop image, the wall features of the wall image, and the ground features of the ground image.

[0068] Step S308: Based on the image features of the surface image, perform semantic segmentation on the image to be identified to obtain the semantic segmentation result of the surface of at least one entity object. The semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object.

[0069] The semantic segmentation results described above may include: semantic information, planar parameters, and planar masks, wherein the semantic information includes: the category of the entity object and the region in which the surface of the entity object is located in the image to be identified.

[0070] The category of the entity object mentioned above can be the type of the entity object itself, that is, the category of the surface of the entity object, such as table, chair, stool, cabinet, refrigerator, etc., but is not limited to this.

[0071] The area where the surface of the aforementioned entity object is located in the image to be recognized can be the position of the candidate box corresponding to that surface in the image to be recognized. This position can be represented by the coordinates (x, y) of the top-left corner of the candidate box and the width and height of the candidate box. Here, the coordinates of the top-left corner can be the coordinates of the top-left corner of the candidate box in a coordinate system established with the top-left corner of the image to be recognized as the origin. The positive direction of the X-axis is to the right from the origin, and the positive direction of the Y-axis is upward from the origin. However, it is not limited to this and can also be the coordinates of other corners of the candidate box. This coordinate system can also be established with other positions in the image to be recognized as the origin, for example, the center position of the image to be recognized as the origin.

[0072] The aforementioned plane parameters can be used to construct the plane equation. The plane equation determines the three-dimensional coordinates of each point in the plane. The plane parameters can be represented as [cosα, cosβ, cosγ, p], and the plane equation can be represented as xcosα + ycosβ + zcosγ = p, but it is not limited to this.

[0073] Whether each pixel in the image to be identified belongs to at least one entity object can be a mask (planar mask) of the entity object's surface. The difference between this and the candidate box is that it is a pixel-level planar segmentation. The shape formed by this segmentation result is no longer a rectangle, but the same as the surface shape of the entity object.

[0074] In an optional embodiment, the area where the surface of the entity object is located in the image to be identified can be obtained based on the position of the candidate box obtained by the above target detection in the image to be identified. Then, image recognition can be performed on the surface image based on the image features of the surface image to obtain the category of the entity object. Furthermore, three-dimensional geometric structure analysis can be performed on the surface based on the image features of the surface image to obtain the planar parameters of the surface of the entity object and the mask of the surface of the entity object.

[0075] In another alternative embodiment, three prediction networks executed in parallel—a semantic prediction network, a parameter prediction network, and a mask prediction network—can be used to perform semantic segmentation of the image to be identified based on the image features of the surface image, obtaining the semantic segmentation result of the surface of at least one entity object. Specifically, the semantic prediction network can perform instance segmentation on the surface of the entity object to obtain the category of the entity object's surface and the precise location of the entity object's surface in the image to be identified; the parameter prediction network can perform three-dimensional geometric structure analysis on the entity object to predict the planar parameters of the entity object's surface; and the mask prediction network can perform instance segmentation on the surface of the entity object to obtain the mask of the entity object's surface.

[0076] It should be noted that the semantic prediction network, parameter prediction network, and mask prediction network mentioned above can directly adopt the networks provided in related technologies, or specific network structures can be set according to actual needs.

[0077] In one optional embodiment, after the user's AR device, VR device, or mobile terminal processes the image to be recognized and obtains the semantic segmentation results of the surfaces of all entity objects in the image, the results can be displayed on a screen for the user to view. In another optional embodiment, after the cloud server processes the image to be recognized and obtains the semantic segmentation results of the surfaces of all entity objects in the image, the results can be sent to the user's AR device, VR device, or mobile terminal and displayed on the screen of the user's AR device, VR device, or mobile terminal for the user to view, such as... Figure 4 As shown, the semantic segmentation results can be displayed in the "Plane Instance Segmentation Display Area".

[0078] For example, taking a 3D fusion rendering scene as an example, the cloud server can perform semantic segmentation on the desktop features of the desktop image, the wall features of the wall image, and the ground features of the ground image, respectively, to obtain the semantic segmentation results of the desktop, the wall, and the ground. Furthermore, based on the three semantic segmentation results, the background image in the room image (e.g., the image of other parts besides the desktop) can be determined, and the background image can be fused with the virtual object to obtain an intelligent fusion result, which can then be displayed to the user.

[0079] Through the above steps, an image to be identified is obtained, wherein the image to be identified contains the surface of at least one entity object; target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; image feature extraction is performed on the surface image to obtain image features of the surface image; semantic segmentation is performed on the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object, thus achieving the technical effect of obtaining the speech information of the image while processing the image. It is easy to note that, when processing the image to be identified, this embodiment of the application can not only perform target detection on the surface of at least one entity object in the image to be identified, but also perform semantic segmentation on the image to be identified based on the image features of different surface images, thereby achieving the technical effect of obtaining plane parameters, plane segmentation mask and semantic information at the same time, and achieving the technical effect of accurately segmenting the surfaces of different objects. This solves the technical problem in related technologies that plane estimation algorithms cannot obtain the semantic information of the plane, resulting in low accuracy of image segmentation. Furthermore, in 3D fusion rendering scenarios, the background image in the image to be identified can be quickly and accurately determined based on the semantic segmentation result, realizing low-cost intelligent recognition of background images and virtual objects, and achieving the effect of reducing the cost of 3D fusion.

[0080] In the above embodiments of this application, semantic segmentation of the image to be identified is performed based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object. This includes: performing semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; performing parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; and performing mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object. The semantic prediction network, parameter prediction network, and mask prediction network are neural network models executed in parallel.

[0081] In the above embodiments of this application, the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified.

[0082] In the above embodiments of this application, the parameter prediction network is composed of fully connected layers.

[0083] In the embodiments described above, the mask prediction network is composed of convolutional layers.

[0084] The semantic prediction network mentioned above can be a Box Head, the parameter prediction network can be a Plane Param Head, and the mask prediction network can be a Mask Head, but it is not limited to these.

[0085] In one optional embodiment, the semantic prediction network Box Head can be composed of two fully connected layers in parallel, but is not limited to this. In this embodiment, two fully connected layers are used as an example. One fully connected layer is used to output the category cls of at least one entity object, and the other fully connected layer is used to output the region [x, yww, h] where the surface of at least one entity object is located in the image to be recognized.

[0086] In another alternative embodiment, the parameter prediction network Plane Param Head may consist of a fully connected layer, but is not limited to this; in this embodiment, a fully connected layer is used as an example. This fully connected layer is used to predict and output the planar parameters [cosα, cosβ, cosγ, p] of the surface of at least one entity object.

[0087] In another optional embodiment, the mask prediction network Mask Head can consist of a 3×3 convolutional layer with a sigmoid activation function. It should be noted that the activation function is not limited to sigmoid, and the convolutional layer is not limited to 3×3; this embodiment uses sigmoid and 3×3 as examples. The convolutional layer outputs whether each pixel in the image to be identified belongs to the surface of at least one entity object, thus obtaining the planar mask M.

[0088] In another alternative embodiment, the image features R of the surface image can be obtained through the Box Head. i Perform instance segmentation to obtain at least one category cls for entity objects, where cls∈R 1×C ,in, This represents a set of real numbers with dimension 1×C, where C is a constant and its specific value can be set according to user needs; and the precise position [x, y, w, h] of the surface of at least one entity object in the image to be recognized, where (x, y) represents the coordinate position of each pixel of the entity object in the planar coordinate system of the image to be recognized, w represents the width of the entity object, and h represents the height of the entity object. T ∈R 1×4 Where T denotes matrix transpose, R 1×4 This represents the set of real numbers with dimension 1×4. Plane Param Heads can be used to modify R... i Perform parameter prediction to obtain the planar parameters of at least one solid object surface, denoted by [cosα, cosβ, cosγ, p].T ∈R 1×4 This indicates that the corresponding plane equation is x cosα + y cosβ + z cosγ = p. Mask Headings can be used to map R... i Performing instance segmentation yields a planar mask M∈R of the surface of at least one entity object. C×28×28 .

[0089] It should be noted that the output resolution of the planar mask M can be 28×28, but it is not limited to this and can also be other resolutions.

[0090] It should be noted that the semantic prediction network, parameter prediction network, and mask prediction network in the embodiments of this application are neural network models that are executed in parallel.

[0091] In the above embodiments of this application, the first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object. The category loss value is determined based on the predicted category and the true category of at least one sample object, and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0092] The aforementioned category loss value can refer to the error between the predicted category of the sample object and the true category of the sample object, or it can be the loss value output by the normalized exponential function (softmax).

[0093] The aforementioned regional loss value can refer to the error between the predicted region where the surface of the sample object is located and the actual region where the surface of the sample object is located, or it can be the loss value output by the L1 loss function.

[0094] The second loss value mentioned above can refer to the error between the predicted plane parameters of the sample object's surface and the true plane parameters of the sample object's surface, and can be calculated using the L1 loss function and L... cossim (n′,n gt The output loss value, n gt Let n′ be the true value of the plane normal, and n′ = [cosα, Cosβ, cosγ].

[0095] The third loss value mentioned above can refer to the error between the predicted plane mask of the sample object's surface and the true plane mask of the sample object, and can be the loss value output after passing through the cross-entropy loss function.

[0096] In one optional embodiment, after the semantic information of the entity object is predicted by the semantic prediction network, the category loss value and the region loss value can be calculated by different loss functions respectively, and the two loss values ​​are summed to obtain the first loss value of the semantic prediction network.

[0097] In another alternative embodiment, after predicting the planar parameters of the sample object's surface using a parameter prediction network, the L1 loss function and L... cossim (n′,n gt The loss function calculates the loss values ​​for the predicted plane parameters and the true plane parameters separately, and then sums the two loss values ​​to obtain the second loss value. The specific formula can be:

[0098] L p =L cossim (n′,n gt )+L1,

[0099] Among them, L p For the second loss value, L cossim (n′,n gt ) represents the loss parameter value, n gt Let n′ be the true value of the plane normal, and let n′ = [cosα, cosβ, cosγ]. Let L1 be the region loss value.

[0100] In another alternative embodiment, after the planar mask of the surface of the entity object is predicted by the mask prediction network, a third loss value can be calculated by the Cross Entropy loss function.

[0101] It should be noted that the three prediction networks can be trained simultaneously, and the sum of the loss function values ​​of the three prediction networks can be used as the final loss function value for end-to-end training of the entire network.

[0102] In the above embodiments of this application, image feature extraction is performed on the surface image to obtain the image features of the surface image, including: performing image feature extraction on the image to be identified using a backbone network to obtain the image features of the image to be identified, wherein the backbone network is a pre-trained neural network model; and performing cropping and deformation operations on the image features of the image to be identified based on the surface image to obtain the image features of the surface image.

[0103] In one optional embodiment, image features are first extracted from the image to be recognized using the backbone, resulting in image features F, such as texture and color features. Secondly, the image features of the surface image are cropped and deformed based on these features, yielding image features R of the surface image. i .

[0104] In the above embodiments of this application, target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object. This includes: performing image feature extraction on the image to be identified using a backbone network to obtain image features of the image to be identified; performing target detection on the surface of at least one entity object in the image to be identified using a candidate region network based on the image features of the image to be identified to obtain a detection result of the surface of at least one entity object, wherein the detection result includes: at least one bounding box and the confidence score of at least one bounding box, the at least one bounding box being used to characterize the position of the surface of at least one entity object in the image to be identified, the confidence score being used to characterize the accuracy of at least one bounding box, and the candidate region network being obtained by training a mask region convolutional neural network; determining a target bounding box from the at least one bounding box based on the confidence score of the at least one bounding box, wherein the confidence score of the target bounding box is greater than a first confidence threshold, and the confidence scores of other bounding boxes in the at least one bounding box besides the target bounding box are less than or equal to the first confidence threshold; and cropping the image to be identified based on the target bounding box to obtain a surface image.

[0105] The aforementioned candidate region network can be an RPN, and the specific network structure can adopt an existing network structure. It needs to be trained before the image to be recognized is processed.

[0106] The aforementioned first confidence threshold can be a threshold set in advance by the user to determine the target bounding box. In response to the fact that the confidence of at least one bounding box is greater than the first confidence threshold, the at least one bounding box can be determined as the target bounding box.

[0107] The target bounding box mentioned above can be a bounding box that can crop out the image to be recognized, that is, the position and size of the target bounding box include the image to be recognized.

[0108] In one optional embodiment, image features are first extracted from the image to be identified using a backbone to obtain image features F, such as texture features and color features. Secondly, a Region Candidate Network (RPN) is used to detect the surface of at least one entity object in the image based on the image features, obtaining at least one bounding box and its confidence level. The RPN is trained on a mask region convolutional neural network. The at least one bounding box represents the position of the surface of at least one entity object in the image to be identified, and the confidence level indicates whether the position of the at least one bounding box is accurate. Then, a target bounding box is determined based on the confidence level of the at least one bounding box and a first confidence threshold. If the confidence level of the at least one bounding box is greater than the first confidence threshold, it is determined that the at least one bounding box is indeed a target bounding box. Finally, the image to be identified is cropped based on the target bounding box to obtain the surface image.

[0109] In the above embodiments of this application, after semantic segmentation of the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object, the method further includes: performing depth prediction on the image to be identified to obtain depth information of the image to be identified; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence of other semantic segmentation results corresponding to other surfaces in the surface of at least one entity object other than the target surface is less than or equal to the second confidence threshold; and performing three-dimensional reconstruction of the target surface based on the target semantic segmentation result and the depth information to obtain a planar point cloud corresponding to the target surface.

[0110] In the above embodiments of this application, the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0111] The depth information of the image to be identified can be the distance of each pixel in the image to be identified from the camera. The depth information can be represented by a depth image with the same resolution as the image to be identified. Each pixel in the depth image is the same as each pixel in the image to be identified. The color intensity of each pixel in the depth image can represent the depth information of each pixel.

[0112] Since a lower confidence score indicates a lower accuracy of semantic segmentation, a second confidence score threshold can be preset to improve the accuracy of 3D reconstruction. For example, the threshold can be 0.5, but it is not limited to this and can be set according to actual needs.

[0113] The activation function mentioned above can be ReLU, but is not limited to this.

[0114] The kernel of the first convolutional layer mentioned above can be 3×3, but is not limited to this.

[0115] The kernel of the second convolutional layer mentioned above can be 1×1, but is not limited to this.

[0116] In one optional embodiment, firstly, a depth estimation network can be used to predict the depth of the image to be identified to obtain the depth information of the image to be identified. Secondly, the confidence of the semantic segmentation result of the surface of the entity object can be confirmed based on a second confidence threshold. When the confidence of the semantic segmentation result of the surface of at least one entity object is greater than the second confidence threshold, the surface of at least one entity object can be determined as the target surface, and the semantic segmentation result of the target surface is the target semantic segmentation result. Finally, the target surface can be reconstructed using the target semantic segmentation result and the depth information to obtain the planar point cloud corresponding to the target surface.

[0117] In another alternative embodiment, the first convolutional layer can be a 3×3 convolutional layer consisting of three ReLU activation functions, and the second convolutional layer can be a 1×1 convolutional layer consisting of one ReLU activation function, but is not limited to these. This results in a depth estimation network composed of multiple convolutional layers.

[0118] In another alternative embodiment, such as Figure 4 As shown, in response to the user's operation of the "Screen Instance Split" button, AR devices, VR devices, or mobile terminals can send the image to be recognized to the cloud server for processing.

[0119] For example, taking a 3D fusion rendering scenario as an example, the cloud server can perform depth prediction on a room image to obtain the depth information of the entire room image. Secondly, it can determine the target surface image from the desktop image, wall image, and ground image based on the semantic segmentation results. For example, the confidence of the surface image can be determined by a second confidence threshold. In response to the desktop image having a confidence greater than the second confidence threshold, the target surface image can be determined to be the desktop image. At the same time, the semantic segmentation result of the desktop image can also be determined to be the target semantic segmentation result. Finally, the cloud server can use the target semantic segmentation result and the depth information of the entire room image to perform 3D reconstruction of the desktop image, obtain the corresponding planar point cloud image of the desktop image, and display it to the user.

[0120] In the above embodiments of this application, depth prediction of the image to be identified is performed to obtain depth information of the image to be identified, including: using a backbone network to perform image feature extraction on the image to be identified to obtain image features of the image to be identified; using a depth estimation network to perform depth prediction of the image to be identified based on the image features of the image to be identified to obtain depth information, wherein the depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network and a mask prediction network.

[0121] In one optional embodiment, image features can first be extracted from the image to be identified using a backbone, resulting in image features F, such as texture features and color features. Then, a depth estimation network, Depth Predictor, can be used to predict the depth of the image to be identified based on its image features, thus obtaining the depth information of the image to be identified, i.e., a depth image.

[0122] In the above embodiments of this application, the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0123] The aforementioned fourth loss value can refer to the error between the three-dimensional coordinates of each pixel point on the surface of the sample object and the true plane coordinates of the sample object's surface. It can be the loss value output by the depth loss function and the L1 loss function.

[0124] In one alternative embodiment, the depth loss function can be L depth2plane The specific calculation formula is as follows:

[0125]

[0126] Where, pi∈R 1×4 Coord represents the true value of the plane parameter of surface i of the solid object. j =[x j y j , z j ,1]∈R 1×4 , [x j y j , z j [] represents the 3D coordinates of the j-th point within the surface i of the entity object, which can be determined by the depth value Z, the 2D pixel coordinates (x, y), and the camera intrinsic parameters K∈R. 3×3 The calculation yields [x, y, z]. T =K-1 ·[X*Z, Y*Z, Z] T .

[0127] In another alternative embodiment, the fourth loss value L corresponding to the depth estimation network d It can be based on the region loss function L1 of the area where the surface of at least one entity object is located and the depth loss function L of the surface of at least one entity object. depth2plane Confirmed, the specific calculation formula is as follows:

[0128] L d =L depth2plane +L1,

[0129] Among them, L d L is the fourth loss value. depth2plane L1 is the depth loss function, and L2 is the region loss function.

[0130] In the above embodiments of this application, based on the target semantic segmentation result and depth information, a three-dimensional reconstruction of the target surface is performed to obtain a planar point cloud corresponding to the target surface, including: determining the target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projection coordinates.

[0131] In one optional embodiment, after determining the target surface, the target depth information corresponding to the target surface in the depth information can first be determined. Then, based on whether each pixel in the image to be identified belongs to the target surface (i.e., the planar mask) and the target depth information (depth map), the target's three-dimensional coordinates [x, y, z] can be calculated using camera imaging formulas. T The calculation formula is as follows:

[0132] [x, y, z] T =K-1·[X*Z, Y*Z, Z] T ,

[0133] Where Z is the value of the planar pixel coordinate (x, y) in the depth information, (x, y) is the coordinate position of the target surface in the planar coordinate system established with the upper left corner of the image to be identified as the origin, and K is a constant. The specific value can be set by the user according to their needs, and is not specifically limited in this embodiment.

[0134] In another alternative embodiment, the projected coordinates [x] of the target's three-dimensional coordinates in the target plane can be obtained based on the target plane parameters of the target surface. p y p , zp ] T The specific calculation formula is as follows:

[0135] [x p y p , z p ] T = [x, y, z] T -[t*cosα, t*cosβ, t*cosγ] T ,

[0136] Among them, t=x*cosα+y*cosβ+z*cosγ.

[0137] The following is combined with Figure 7 A preferred embodiment of this application will be described in detail below. Figure 7 This is a schematic diagram of an optional image processing method according to an embodiment of this application, such as... Figure 7 As shown, the method includes the following steps:

[0138] S1, input a three-channel image to a backbone network to extract image features F, and send them to the instance segmentation network and the depth estimation network (Depth Head) respectively for planar mask and parameter estimation and depth map estimation (Depthpredictor). In this embodiment, the image format supported is not limited and can be extended to video scenes. The instance segmentation algorithm can also be replaced by a panoramic segmentation algorithm, and the number of categories is not limited.

[0139] S2, the image features F extracted in step S1 are input into a depth estimation network to predict the depth information of the image. In this network, the value (Map) of each pixel in the depth map is >0. The depth network consists of three 3x3 convolutional layers with ReLU as the activation function and one 1x1 convolutional layer with ReLU as the activation function. The value of Map is greater than 0. Convolutional neural networks can use a variety of different structures, such as adjusting the number of convolutional module layers. In this embodiment, it is not limited to a specific structure.

[0140] S3, a Mask-RCNN pre-trained Region Proposal Network (RPN) is used to extract multiple candidate boxes (Proposals), and the feature map R of each candidate box is calculated from the image features F extracted in step S1 through cropping and warping operations. i The candidate region extraction network can use different designs, and in this embodiment, it is not limited to using a certain fixed structure design.

[0141] S4. Each candidate box feature is processed by three different prediction networks: a semantic prediction network (BoxHead), a parameter prediction network (Plane Param Head), and a mask prediction network (Mask Head). The BoxHead consists of two fully connected layers in parallel, each responsible for predicting the class cls∈R of the input candidate box. 1×C And the precise candidate bounding box representation [x, y, w, h] T ∈R 1×4 This refers to the coordinates of the top-left corner (x, y) and the width (w) and height (h) of the candidate box; the Plane ParamHead consists of one fully connected layer, responsible for predicting the parameters of the plane within the input candidate box, denoted by [cosα, cosβ, cosγ, p]. T ∈R 1×4 The corresponding plane equation is xcosα + ycosβ + z cosγ = p; the Mask Head consists of a 3×3 convolutional layer with Sigmoid as the activation function, responsible for predicting the planar segmentation mask M∈R for each category. C×28×28 In this context, the value of each pixel in the mask image is 0 to 1, and the output resolution of the Mask Head can be changed and is not limited to 28×28.

[0142] S5. During training, different prediction heads use the loss function corresponding to the task. Multiple loss functions are summed to obtain the final loss function for end-to-end training of the entire network. Specifically, the Box Head's category output cls uses softmax loss, and the candidate box representation [x, y, w, h] uses L1 loss; the Plane Param Head uses both L1 and Cosine Simulary losses. p =L cossim (n′,n gt )+L1,n gt For the true values ​​of the plane normal, n′=[cosα, cosβ, cosγ]; the Mask Head uses Cross Entropy loss; the Depth Head uses L1 and Depth2Plane losses, where L... d =L depth2plane +L1. The Depth2Plane loss is calculated as follows: in Let i be the true value of the plane parameter of plane i. [x j y j , z j [] represents the 3D coordinates of the j-th point in plane i, which can be obtained from the point's depth value Z, 2D pixel coordinates (X, Y), and camera intrinsic parameters. The calculation yields [x, y, z]. T =K -1 ·[X*Z, Y*Z, Z] T .

[0143] S6, Plane Reconstruction Process in the Testing Phase: In the testing phase, by setting a threshold, planar instances with a confidence level greater than 0.5 can be extracted and output. Each instance includes a bounding box result [x, y, h, w], a plane mask, and plane parameters [cosα, cosβ, cosγ, p]. T And a Depthmap. Based on the plane mask and its corresponding depth, the 3D coordinates [x, y, z] of the plane can be calculated using the camera imaging formula. T =K -1 ·[X*Z, Y*Z, Z] T Where Z is the value of the 2D pixel coordinates (X, Y) in the Depthmap. Based on the plane parameters, the projection coordinates of the 3D coordinate point onto the plane [x, y] can be obtained. p y p , z p ] T This data is then used as the final planar point cloud to complete the planar reconstruction. The formula is [x...]. p y p , z p ] T = [x, y, z] T -[t*cosα, t*cosβ, t*cosγ] T , where t=x*cosα+y*cosβ+z*cosγ.

[0144] It should be noted that, by training and testing on ScanNet V2, the models in this application (including Ours (baseline) and Ours (+loss)) outperform the two related methods (including PlaneNet and PlaneRCNN) mentioned above in both depth estimation and plane detection evaluation metrics (DepthMetrics and PlaneMetrics, respectively), as shown in Table 1:

[0145] Table 1

[0146]

[0147] Here, Rel (Relative error) represents the relative error; the smaller the relative error, the more accurate the depth estimation. RMSE (Root Mean Square Error) represents the root mean square error; the smaller the root mean square error, the more accurate the depth estimation. δ iThe threshold precision is represented by i, which is 1, 2, or 3. Higher threshold precision indicates more accurate depth estimation. VI (Variation of Information) represents the information difference index. The smaller the index value, the better the planar clustering result matches the actual situation. RI (Rand Index) represents the Rand index. The larger the value, the better the planar clustering result matches the actual situation. SC (Segmentation Covering) represents the segmentation coverage. The larger the segmentation coverage value, the better the planar clustering result matches the actual situation.

[0148] As shown in the table above, the planar segmentation and depth estimation results of this application are superior to those of PlaneNet and Plane RCNN.

[0149] It should be noted that, through instance segmentation or panoptic segmentation design, the model in this application embodiment can output planar semantics and is not limited in the number of planes; through Plane Param Head design, this application embodiment can predict more accurate planar parameters; by combining single-image instance segmentation and depth estimation tasks for joint training, compared to PlaneRCNN which requires video data for training, the model in this application embodiment only requires image data for training, which greatly reduces the training cost.

[0150] In one optional embodiment, when the image to be identified contains a table, the desktop image is first obtained by target detection in the image to be identified through the backbone network. Secondly, the image features F of the image to be identified are extracted by the backbone network, and the image features of the desktop image are obtained by cropping and warping the image to be identified based on the desktop image (Proposal) through the RPN network. Then, semantic segmentation, parameter prediction, and mask prediction are performed on the image to be identified by three parallel prediction networks: Box Head, Plane Param Head, and Mask Head.

[0151] Then, during model training, three prediction networks are trained using corresponding loss functions. The first loss value cls and [x, y, w, h] for the desktop image are obtained after training the Box Head; the second loss value [cosα, cosβ, cosγ] / p for the desktop image is obtained after training the Plane Param Head; and the third loss value M∈R for the desktop image is obtained after training the Mask Head. C×28×28 The range of the map is 0 to 1 and the activation function is Sigmoid.

[0152] It should be noted that after obtaining the semantic segmentation results of the desktop, the depth of the image to be recognized can be predicted by the DepthHead depth estimation network to obtain the depth information of the image to be recognized. The depth information includes a resolution of 1×480×640, a map range greater than 0, and an activation function of ReLU.

[0153] Finally, the desktop is reconstructed using the target semantic segmentation results and depth information to obtain the planar point cloud of the desktop.

[0154] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0156] Example 2

[0157] According to an embodiment of this application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0158] Figure 8 This is a flowchart of an image processing method according to Embodiment 2 of this application, as follows: Figure 8 As shown, the method may include the following steps:

[0159] Step S802: In response to an input command applied to the operation interface, display the image to be recognized on the operation interface, wherein the image to be recognized contains the surface of at least one entity object.

[0160] The aforementioned user interface may include, but is not limited to, the user interface of a mobile terminal or the user interface of a server. Specifically, it may be the user interface of a laptop, a mobile phone, or a tablet computer.

[0161] The input instructions mentioned above may include, but are not limited to, voice input and text input. For example, a mobile terminal or server may receive a user's text input instruction "image recognition is required," or a voice receiving device on the mobile terminal or server may receive a user's voice instruction "image recognition is required," but these are not limited to these.

[0162] In one alternative embodiment, such as Figure 4 As shown, the operation interface may include, but is not limited to, the following functions: an image upload area, a "Plane Instance Segmentation" button, a plane instance segmentation display area, and a "Shoot" button. In response to receiving the user's operation of the "Shoot" button on the operation interface, the mobile terminal can take a picture of the area where the user is located to obtain the image to be recognized. The mobile terminal can also display the image to be recognized on the operation interface for the user to view in order to determine whether the image to be recognized was captured correctly. The image to be recognized contains the surface of at least one entity object, such as a desktop, wall, or ground.

[0163] Step S804: In response to the instance segmentation command applied to the operation interface, the semantic segmentation result of the surface of at least one entity object is displayed on the operation interface. The semantic segmentation result is obtained by semantically segmenting the image to be identified based on the image features of the surface image of the surface of at least one entity object. The semantic segmentation result includes: the category of at least one entity object, the region where the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object. The surface image is obtained by performing target detection on the surface of at least one entity object in the image to be identified, and the image features of the surface image are obtained by performing image feature extraction on the surface image.

[0164] The instance segmentation instructions mentioned above may include, but are not limited to: semantic segmentation of surface images, parameter prediction of surface images, mask prediction of surface images, and depth prediction of surface images.

[0165] In one alternative embodiment, such as Figure 4As shown, after acquiring the image to be recognized, the user can drag the image to be recognized into the dashed box and press the "Plane Instance Segmentation" button. The AR device, VR device, or mobile device can then upload the image to be recognized to the cloud server for processing. In response to the cloud server completing the processing of the image to be recognized based on the instance segmentation instruction, the AR device, VR device, or mobile device can display the plane instance segmentation result in the plane instance segmentation display area for the user to view.

[0166] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0167] Example 3

[0168] According to the embodiments of this application, an image processing method applicable to virtual reality scenarios such as virtual reality (VR) devices and augmented reality (AR) devices is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0169] Figure 9 This is a flowchart of an image processing method according to Embodiment 3 of this application. Figure 9 As shown, the method may include the following steps:

[0170] Step S902: Display the image to be identified on the presentation screen of the virtual reality (VR) device or augmented reality (AR) device, wherein the image to be identified contains the surface of at least one entity object.

[0171] In one alternative embodiment, such as Figure 4 As shown, when a user wears a VR or AR device, the AR / VR device will take a picture of the environment seen by the user's eyes (i.e., the image to be recognized) in response to the user's operation of the "shoot" button on the operation interface, and display the image to be recognized to the user in the display area of ​​the device. The image to be recognized contains the surface of at least one entity object.

[0172] In another alternative embodiment, when a user wears a VR or AR device in a room, in response to the user's "shooting" operation on the VR or AR device, the VR or AR device will take a picture of the surrounding environment. For example, the VR or AR device can take a picture of the environment containing a table that the user sees, and obtain an image containing the table (i.e., the image to be identified). After obtaining the image to be identified, the VR or AR device can also display the captured image containing the table in the display area of ​​the device to show the user, wherein the image to be identified contains the surface of at least one entity object.

[0173] Step S904: Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object.

[0174] Step S906: Perform image feature extraction on the surface image to obtain the image features of the surface image.

[0175] Step S908: Semantic segmentation is performed on the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object. The semantic segmentation result is obtained by semantic segmentation of the image to be identified based on the image features of the surface image of the surface of at least one entity object. The semantic segmentation result includes: the category of at least one entity object, the region where the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object. The surface image is obtained by target detection on the surface of at least one entity object in the image to be identified, and the image features of the surface image are obtained by performing image feature extraction on the surface image.

[0176] The instance segmentation instructions mentioned above may include, but are not limited to: semantic segmentation of surface images, parameter prediction of surface images, mask prediction of surface images, and depth prediction of surface images.

[0177] In one alternative embodiment, such as Figure 4 As shown, after acquiring the image to be recognized, the user can drag the image to be recognized into the dashed box and press the "Plane Instance Segmentation" button. The AR device or VR device can then upload the image to be recognized to the cloud server for processing. In response to the cloud server completing the processing of the image to be recognized based on the instance segmentation instruction, the AR device, VR device, or mobile device can display the plane instance segmentation result in the plane instance segmentation display area for the user to view.

[0178] Step S910: Drive the VR device or AR device to display the semantic segmentation result of the surface of at least one entity object.

[0179] In one optional embodiment, after the cloud server performs semantic segmentation on the image, it returns the semantic segmentation result to the AR or VR device. The AR or VR device then displays the semantic segmentation result to the user in the "planar instance segmentation display area".

[0180] In the above embodiments of this application, semantic segmentation of the image to be identified is performed based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object. This includes: performing semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; performing parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; and performing mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object. The semantic prediction network, parameter prediction network, and mask prediction network are neural network models executed in parallel.

[0181] In the above embodiments of this application, the first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object. The category loss value is determined based on the predicted category and the true category of at least one sample object, and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0182] In the above embodiments of this application, the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified.

[0183] In the above embodiments of this application, the parameter prediction network is composed of fully connected layers.

[0184] In the embodiments described above, the mask prediction network is composed of convolutional layers.

[0185] In the above embodiments of this application, image feature extraction is performed on the surface image to obtain the image features of the surface image, including: performing image feature extraction on the image to be identified using a backbone network to obtain the image features of the image to be identified, wherein the backbone network is a pre-trained neural network model; and performing cropping and deformation operations on the image features of the image to be identified based on the surface image to obtain the image features of the surface image.

[0186] In the above embodiments of this application, target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object. This includes: using a backbone network to perform image feature extraction on the image to be identified to obtain image features of the image to be identified; using a candidate region network to perform target detection on the surface of at least one entity object in the image to be identified based on the image features of the image to be identified to obtain a detection result of the surface of at least one entity object, wherein the detection result includes: at least one bounding box and the confidence score of at least one bounding box, the at least one bounding box being used to characterize the position of the surface of at least one entity object in the image to be identified, the confidence score being used to characterize the accuracy of at least one bounding box, and the candidate region network being obtained by training a mask region convolutional neural network; determining a target bounding box from the at least one bounding box based on the confidence score of the at least one bounding box, wherein the confidence score of the target bounding box is greater than a first confidence threshold, and the confidence scores of other bounding boxes in the at least one bounding box besides the target bounding box are less than or equal to the first confidence threshold; and cropping the image to be identified based on the target bounding box to obtain a surface image.

[0187] In the above embodiments of this application, after semantic segmentation of the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object, the method further includes: performing depth prediction on the image to be identified to obtain depth information of the image to be identified; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence level of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence level of other semantic segmentation results corresponding to other surfaces of at least one entity object besides the target surface is less than or equal to the second confidence threshold. Based on the target semantic segmentation result and the depth information, three-dimensional reconstruction is performed on the target surface to obtain a planar point cloud corresponding to the target surface.

[0188] In the above embodiments of this application, depth prediction of the image to be identified is performed to obtain depth information of the image to be identified, including: using a backbone network to perform image feature extraction on the image to be identified to obtain image features of the image to be identified; using a depth estimation network to perform depth prediction of the image to be identified based on the image features of the image to be identified to obtain depth information, wherein the depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network and a mask prediction network.

[0189] In the above embodiments of this application, the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0190] In the above embodiments of this application, the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0191] In the above embodiments of this application, based on the target semantic segmentation result and depth information, a three-dimensional reconstruction of the target surface is performed to obtain a planar point cloud corresponding to the target surface, including: determining the target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projection coordinates.

[0192] Optionally, in this embodiment, the image processing method described above can be applied to a hardware environment consisting of a server and a virtual reality device. The image to be identified is displayed on the screen of the virtual reality (VR) device or augmented reality (AR) device. The server can be a server corresponding to a media file operator. The network mentioned above includes, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), or a local area network (LAN). The virtual reality device is not limited to, for example, a virtual reality headset, virtual reality glasses, or a standalone virtual reality device.

[0193] Optionally, the virtual reality device includes: a memory, a processor, and a transmission device. The memory stores an application that can be used to perform: displaying an image to be identified on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image to be identified contains the surface of at least one entity object; performing object detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object, wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object; and driving the VR device or AR device to display the semantic segmentation result of the surface of at least one entity object.

[0194] It should be noted that the image processing method described above in this embodiment, when applied to VR or AR devices, may include... Figure 3The method of the illustrated embodiment is intended to drive a VR device or AR device to display the semantic segmentation results of the surface of at least one entity object.

[0195] Optionally, the processor in this embodiment can invoke the application stored in the memory via the transmission device to perform the above steps. The transmission device can receive media files sent by the server via a network, and can also be used for data transmission between the processor and the memory.

[0196] Optionally, in a virtual reality device, there is a head-mounted display with eye tracking. The screen in the HMD is used to display the video footage. The eye tracking module in the HMD is used to acquire the real-time movement trajectory of the user's eyes. The tracking system is used to track the user's position and movement information in real three-dimensional space. The computing and processing unit is used to acquire the user's real-time position and movement information from the tracking system and calculate the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of vision orientation in the virtual three-dimensional space.

[0197] In this embodiment, the virtual reality device can be connected to a terminal, and the terminal and the server are connected via a network. The virtual reality device is not limited to virtual reality headsets, virtual reality glasses, virtual reality all-in-one machines, etc., and the terminal is not limited to PCs, mobile phones, tablets, etc. The server can be a server corresponding to a media file operator, and the network includes, but is not limited to, wide area networks, metropolitan area networks, or local area networks.

[0198] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0199] Example 4

[0200] According to an embodiment of this application, an embodiment of an image processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0201] Figure 10 This is a flowchart of an image processing method according to Embodiment 4 of this application, as follows: Figure 10 As shown, the method may include the following steps:

[0202] Step S1002: Obtain the image to be recognized by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the image to be recognized, and the image to be recognized contains the surface of at least one entity object.

[0203] The first interface mentioned above can be the interface used by the user's AR device, VR device, or mobile terminal to send data to the cloud server. The image to be recognized can be uploaded to the cloud server through the first interface.

[0204] In one alternative embodiment, the AR device, VR device, or mobile terminal may, in response to a user's upload operation, upload the image to be recognized to the cloud server via a first interface.

[0205] Step S1004: Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object.

[0206] Step S1006: Perform image feature extraction on the surface image to obtain the image features of the surface image.

[0207] Step S1008: Semantic segmentation is performed on the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object. The semantic segmentation result is obtained by semantic segmentation of the image to be identified based on the image features of the surface image of the surface of at least one entity object. The semantic segmentation result includes: the category of at least one entity object, the region where the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object. The surface image is obtained by target detection on the surface of at least one entity object in the image to be identified, and the image features of the surface image are obtained by performing image feature extraction on the surface image.

[0208] The instance segmentation instructions mentioned above may include, but are not limited to: semantic segmentation of surface images, parameter prediction of surface images, mask prediction of surface images, and depth prediction of surface images.

[0209] Step S1010: Output the semantic segmentation result of the surface of at least one entity object by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the semantic segmentation result of the surface of at least one entity object.

[0210] The second interface mentioned above can be an interface for the cloud server to send data to the client. Through the second interface, the semantic segmentation result of the surface of at least one entity object can be sent to the user's AR device, VR device or mobile terminal.

[0211] In one optional embodiment, after the cloud server has completed semantic segmentation, the semantic segmentation result can be returned to the AR device, VR device, or mobile terminal through the second interface.

[0212] In the above embodiments of this application, semantic segmentation of the image to be identified is performed based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object. This includes: performing semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; performing parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; and performing mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object. The semantic prediction network, parameter prediction network, and mask prediction network are neural network models executed in parallel.

[0213] In the above embodiments of this application, the first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object. The category loss value is determined based on the predicted category and the true category of at least one sample object, and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0214] In the above embodiments of this application, the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified.

[0215] In the above embodiments of this application, the parameter prediction network is composed of fully connected layers.

[0216] In the embodiments described above, the mask prediction network is composed of convolutional layers.

[0217] In the above embodiments of this application, image feature extraction is performed on the surface image to obtain the image features of the surface image, including: performing image feature extraction on the image to be identified using a backbone network to obtain the image features of the image to be identified, wherein the backbone network is a pre-trained neural network model; and performing cropping and deformation operations on the image features of the image to be identified based on the surface image to obtain the image features of the surface image.

[0218] In the above embodiments of this application, target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object. This includes: performing image feature extraction on the image to be identified using a backbone network to obtain image features of the image to be identified; performing target detection on the surface of at least one entity object in the image to be identified using a candidate region network based on the image features of the image to be identified to obtain a detection result of the surface of at least one entity object, wherein the detection result includes: at least one bounding box and the confidence score of at least one bounding box, the at least one bounding box being used to characterize the position of the surface of at least one entity object in the image to be identified, the confidence score being used to characterize the accuracy of at least one bounding box, and the candidate region network being obtained by training a mask region convolutional neural network; determining a target bounding box from the at least one bounding box based on the confidence score of the at least one bounding box, wherein the confidence score of the target bounding box is greater than a first confidence threshold, and the confidence scores of other bounding boxes in the at least one bounding box besides the target bounding box are less than or equal to the first confidence threshold; and cropping the image to be identified based on the target bounding box to obtain a surface image.

[0219] In the above embodiments of this application, after semantic segmentation of the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of at least one entity object, the method further includes: performing depth prediction on the image to be identified to obtain depth information of the image to be identified; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence level of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence level of other semantic segmentation results corresponding to other surfaces of at least one entity object besides the target surface is less than or equal to the second confidence threshold. Based on the target semantic segmentation result and the depth information, three-dimensional reconstruction is performed on the target surface to obtain a planar point cloud corresponding to the target surface.

[0220] In the above embodiments of this application, depth prediction of the image to be identified is performed to obtain depth information of the image to be identified, including: using a backbone network to perform image feature extraction on the image to be identified to obtain image features of the image to be identified; using a depth estimation network to perform depth prediction of the image to be identified based on the image features of the image to be identified to obtain depth information, wherein the depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network and a mask prediction network.

[0221] In the above embodiments of this application, the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0222] In the above embodiments of this application, the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0223] In the above embodiments of this application, based on the target semantic segmentation result and depth information, a three-dimensional reconstruction of the target surface is performed to obtain a planar point cloud corresponding to the target surface, including: determining the target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projection coordinates.

[0224] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0225] Example 5

[0226] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 11 This is a schematic diagram of an image processing apparatus according to Embodiment 5 of this application, as shown below. Figure 11 As shown, the device includes: an acquisition module 1102, a detection module 1104, a feature extraction module 1106, and a semantic segmentation module 1108.

[0227] The acquisition module acquires an image to be identified, wherein the image to be identified contains the surface of at least one entity object; the detection module performs target detection on the surface of at least one entity object in the image to be identified, thereby obtaining a surface image of the surface of at least one entity object; the feature extraction module performs image feature extraction on the surface image, thereby obtaining image features of the surface image; the semantic segmentation module performs semantic segmentation on the image to be identified based on the image features of the surface image, thereby obtaining a semantic segmentation result of the surface of at least one entity object; wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object.

[0228] It should be noted that the acquisition module, detection module, feature extraction module, and semantic segmentation module mentioned above correspond to steps S502 to S508 in Embodiment 1. The instances and application scenarios implemented by the two modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the AR / VR device provided in Embodiment 1.

[0229] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0230] In the above embodiments of this application, the semantic segmentation module includes: a semantic segmentation unit, a parameter prediction unit, and a mask prediction unit.

[0231] The semantic segmentation unit is used to perform semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; the parameter prediction unit is used to perform parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; the mask prediction unit is used to perform mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein, the semantic prediction network, the parameter prediction network and the mask prediction network are neural network models that are executed in parallel.

[0232] In the above embodiments of this application, the first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object. The category loss value is determined based on the predicted category and the true category of at least one sample object, and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0233] In the above embodiments of this application, the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified.

[0234] In the above embodiments of this application, the parameter prediction network is composed of fully connected layers.

[0235] In the embodiments described above, the mask prediction network is composed of convolutional layers.

[0236] In the above embodiments of this application, the feature extraction module includes: a first image feature extraction unit and a processing unit.

[0237] The first image feature extraction unit is used to extract image features from the image to be identified using a backbone network, where the backbone network is a pre-trained neural network model; the processing unit is used to perform cropping and deformation operations on the image features of the image to be identified based on the surface image, so as to obtain the image features of the surface image.

[0238] In the above embodiments of this application, the detection module includes: a second image feature extraction unit, a target detection unit, a bounding box determination unit, and an image cropping unit.

[0239] The second image feature extraction unit is used to extract image features from the image to be identified using a backbone network, thereby obtaining image features of the image to be identified. The target detection unit is used to perform target detection on the surface of at least one entity object in the image to be identified based on the image features of the image to be identified using a candidate region network, thereby obtaining the detection result of the surface of at least one entity object. The detection result includes at least one bounding box and the confidence score of at least one bounding box. The at least one bounding box is used to characterize the position of the surface of at least one entity object in the image to be identified, and the confidence score is used to characterize the accuracy of the at least one bounding box. The candidate region network is obtained by training a mask region convolutional neural network. The bounding box determination unit is used to determine the target bounding box from the at least one bounding box based on the confidence score of the at least one bounding box. The confidence score of the target bounding box is greater than a first confidence score threshold, and the confidence scores of other bounding boxes in the at least one bounding box, excluding the target bounding box, are less than or equal to the first confidence score threshold. The image cropping unit is used to crop the image to be identified based on the target bounding box to obtain a surface image.

[0240] In the above embodiments of this application, the device further includes: a depth prediction module, a determination module, and a three-dimensional reconstruction module.

[0241] The depth prediction module is used to predict the depth of the image to be recognized and obtain the depth information of the image to be recognized. The determination module is used to determine the target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence of other semantic segmentation results corresponding to other surfaces in the surface of at least one entity object other than the target surface is less than or equal to the second confidence threshold. The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the target surface based on the target semantic segmentation result and the depth information to obtain the planar point cloud corresponding to the target surface.

[0242] In the above embodiments of this application, the depth prediction module includes: a feature extraction unit and a depth prediction unit.

[0243] The feature extraction unit is used to extract image features from the image to be identified using the backbone network, and obtain the image features of the image to be identified. The depth prediction unit is used to predict the depth of the image to be identified based on the image features of the image to be identified using the depth estimation network, and obtain depth information. The depth estimation network is a neural network model obtained by jointly training sample images with semantic prediction network, parameter prediction network and mask prediction network.

[0244] In the above embodiments of this application, the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0245] In the above embodiments of this application, the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0246] In the above embodiments of this application, the three-dimensional reconstruction module includes: a first determining unit, a second determining unit, a third determining unit, and a generating unit.

[0247] The first determining unit is used to determine the target depth information corresponding to the target surface in the depth information; the second determining unit is used to determine the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; the third determining unit is used to determine the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and the generating unit is used to generate a planar point cloud based on the projection coordinates.

[0248] Example 6

[0249] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 12 This is a schematic diagram of an image processing apparatus according to Embodiment 6 of this application, as shown below. Figure 12 As shown, the device includes: a first display module 1202 and a second display module 1204.

[0250] The first display module is used to respond to input commands applied to the operation interface and display an image to be recognized on the operation interface, wherein the image to be recognized contains the surface of at least one entity object; the second display module is used to respond to instance segmentation commands applied to the operation interface and display the semantic segmentation result of the surface of at least one entity object on the operation interface, wherein the semantic segmentation result is obtained by semantically segmenting the image to be recognized based on the image features of the surface image of the surface of at least one entity object, and the semantic segmentation result includes: the category of at least one entity object, the region in the image to be recognized where the surface of at least one entity object is located, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be recognized belongs to the surface of at least one entity object, the surface image is obtained by target detection on the surface of at least one entity object in the image to be recognized, and the image features of the surface image are obtained by performing image feature extraction on the surface image.

[0251] It should be noted that the first display module 1202 and the second display module 1204 mentioned above correspond to steps S802 to S804 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors. The above modules can also be part of the device and run in the AR / VR device provided in Embodiment 1.

[0252] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0253] Example 7

[0254] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 13 This is a schematic diagram of an image processing apparatus according to Embodiment 7 of this application, as shown below. Figure 13 As shown, the device includes: a first display module 1302, a detection module 1304, a feature extraction module 1306, a semantic segmentation module 1309, and a second display module 1310.

[0255] The first display module is used to display the image to be identified on the presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image to be identified contains the surface of at least one entity object; the detection module is used to perform target detection on the surface of at least one entity object in the image to be identified, thereby obtaining a surface image of the surface of at least one entity object; the feature extraction module is used to perform image feature extraction on the surface image, thereby obtaining the image features of the surface image; the semantic segmentation module is used to perform semantic segmentation on the image to be identified based on the image features of the surface image, thereby obtaining the semantic segmentation result of the surface of at least one entity object, wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object; the second display module is used to drive the VR device or AR device to display the semantic segmentation result of the surface of at least one entity object.

[0256] It should be noted that the first display module, detection module, feature extraction module, semantic segmentation module, and second display module mentioned above correspond to steps S902 to S910 in Embodiment 3. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the AR / VR device provided in Embodiment 1.

[0257] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0258] In the above embodiments of this application, the semantic segmentation module includes: a semantic segmentation unit, a parameter prediction unit, and a mask prediction unit.

[0259] The semantic segmentation unit is used to perform semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; the parameter prediction unit is used to perform parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; the mask prediction unit is used to perform mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein, the semantic prediction network, the parameter prediction network and the mask prediction network are neural network models that are executed in parallel.

[0260] In the above embodiments of this application, the first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object. The category loss value is determined based on the predicted category and the true category of at least one sample object, and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0261] In the above embodiments of this application, the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified.

[0262] In the above embodiments of this application, the parameter prediction network is composed of fully connected layers.

[0263] In the embodiments described above, the mask prediction network is composed of convolutional layers.

[0264] In the above embodiments of this application, the feature extraction module includes: a first image feature extraction unit and a processing unit.

[0265] The first image feature extraction module is used to extract image features from the image to be identified using a backbone network, where the backbone network is a pre-trained neural network model. The processing unit is used to perform cropping and deformation operations on the image features of the image to be identified based on the surface image, so as to obtain the image features of the surface image.

[0266] In the above embodiments of this application, the detection module includes: a second image feature extraction unit, a target detection unit, a bounding box determination unit, and an image cropping unit.

[0267] The second image feature extraction unit uses a backbone network to extract image features from the image to be identified, obtaining image features of the image to be identified. The target detection unit uses a candidate region network to perform target detection on the surface of at least one entity object in the image to be identified based on the image features of the image to be identified, obtaining the detection result of the surface of at least one entity object. The detection result includes at least one bounding box and the confidence of at least one bounding box. The at least one bounding box is used to characterize the position of the surface of at least one entity object in the image to be identified, and the confidence is used to characterize the accuracy of at least one bounding box. The candidate region network is obtained by training a mask region convolutional neural network. The bounding box determination unit determines a target bounding box from at least one bounding box based on the confidence of at least one bounding box. The confidence of the target bounding box is greater than a first confidence threshold, and the confidence of other bounding boxes in at least one bounding box, excluding the target bounding box, is less than or equal to the first confidence threshold. The image cropping unit crops the image to be identified based on the target bounding box to obtain a surface image.

[0268] In the above embodiments of this application, the device further includes: a depth prediction module, a determination module, and a three-dimensional reconstruction module.

[0269] The depth prediction module is used to predict the depth of the image to be recognized and obtain the depth information of the image to be recognized. The determination module is used to determine the target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence of other semantic segmentation results corresponding to other surfaces in the surface of at least one entity object other than the target surface is less than or equal to the second confidence threshold. The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the target surface based on the target semantic segmentation result and the depth information to obtain the planar point cloud corresponding to the target surface.

[0270] In the above embodiments of this application, the depth prediction module includes: a feature extraction unit and a depth prediction unit.

[0271] The feature extraction unit is used to extract image features from the image to be identified using the backbone network, and obtain the image features of the image to be identified. The depth prediction unit is used to predict the depth of the image to be identified based on the image features of the image to be identified using the depth estimation network, and obtain depth information. The depth estimation network is a neural network model obtained by jointly training sample images with semantic prediction network, parameter prediction network and mask prediction network.

[0272] In the above embodiments of this application, the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0273] In the above embodiments of this application, the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0274] In the above embodiments of this application, the three-dimensional reconstruction module includes: a first determining unit, a second determining unit, a third determining unit, and a generating unit.

[0275] The first determining unit is used to determine the target depth information corresponding to the target surface in the depth information; the second determining unit is used to determine the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; the third determining unit is used to determine the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and the generating unit is used to generate a planar point cloud based on the projection coordinates.

[0276] Example 8

[0277] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided. Figure 14 This is a schematic diagram of an image processing apparatus according to Embodiment 8 of this application, as shown below. Figure 14 As shown, the device includes: an acquisition module 1402, a detection module 1404, a feature extraction module 1406, a semantic segmentation module 1408, and an output module 1410.

[0278] The system comprises the following modules: an acquisition module for acquiring an image to be identified by calling a first interface, wherein the first interface includes a first parameter whose value is the image to be identified, and the image to be identified contains the surface of at least one entity object; a detection module for performing target detection on the surface of at least one entity object in the image to be identified, thereby obtaining a surface image of the surface of at least one entity object; a feature extraction module for performing image feature extraction on the surface image, thereby obtaining image features of the surface image; a semantic segmentation module for performing semantic segmentation on the image to be identified based on the image features of the surface image, thereby obtaining a semantic segmentation result of the surface of at least one entity object, wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object; and an output module for outputting the semantic segmentation result of the surface of at least one entity object by calling a second interface, wherein the second interface includes a second parameter whose value is the semantic segmentation result of the surface of at least one entity object.

[0279] It should be noted that the acquisition module, detection module, feature extraction module, semantic segmentation module, and output module mentioned above correspond to steps S1002 to S1010 in Embodiment 4. The instances and application scenarios implemented by the two modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1 above. It should be noted that the above modules or units can be hardware components or software components stored in memory and processed by one or more processors.

[0280] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0281] In the above embodiments of this application, the semantic segmentation module includes: a semantic segmentation unit, a parameter prediction unit, and a mask prediction unit.

[0282] The semantic segmentation unit is used to perform semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; the parameter prediction unit is used to perform parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; the mask prediction unit is used to perform mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein, the semantic prediction network, the parameter prediction network and the mask prediction network are neural network models that are executed in parallel.

[0283] In the above embodiments of this application, the first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object. The category loss value is determined based on the predicted category and the true category of at least one sample object, and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0284] In the above embodiments of this application, the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified.

[0285] In the above embodiments of this application, the parameter prediction network is composed of fully connected layers.

[0286] In the embodiments described above, the mask prediction network is composed of convolutional layers.

[0287] In the above embodiments of this application, the feature extraction module includes: a first image feature extraction unit and a processing unit.

[0288] The first image feature extraction unit is used to extract image features from the image to be identified using a backbone network, where the backbone network is a pre-trained neural network model; the processing unit is used to perform cropping and deformation operations on the image features of the image to be identified based on the surface image, so as to obtain the image features of the surface image.

[0289] In the above embodiments of this application, the detection module includes: a second image feature extraction unit and a target detection unit.

[0290] The second image feature extraction unit is used to extract image features from the image to be identified using a backbone network, thereby obtaining image features of the image to be identified. The target detection unit is used to perform target detection on the surface of at least one entity object in the image to be identified based on the image features of the image to be identified using a candidate region network, thereby obtaining the detection result of the surface of at least one entity object. The detection result includes at least one bounding box and the confidence score of at least one bounding box. The at least one bounding box is used to characterize the position of the surface of at least one entity object in the image to be identified, and the confidence score is used to characterize the accuracy of the at least one bounding box. The candidate region network is obtained by training a mask region convolutional neural network. Based on the confidence score of at least one bounding box, a target bounding box is determined from at least one bounding box, wherein the confidence score of the target bounding box is greater than a first confidence threshold, and the confidence scores of other bounding boxes in at least one bounding box besides the target bounding box are less than or equal to the first confidence threshold. The image to be identified is cropped based on the target bounding box to obtain a surface image.

[0291] In the above embodiments of this application, the device further includes: a depth prediction module, a determination module, and a three-dimensional reconstruction module.

[0292] The depth prediction module is used to predict the depth of the image to be recognized and obtain the depth information of the image to be recognized. The determination module is used to determine the target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence of other semantic segmentation results corresponding to other surfaces in the surface of at least one entity object other than the target surface is less than or equal to the second confidence threshold. The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the target surface based on the target semantic segmentation result and the depth information to obtain the planar point cloud corresponding to the target surface.

[0293] In the above embodiments of this application, the depth prediction module includes: a feature extraction unit and a depth prediction unit.

[0294] The feature extraction unit is used to extract image features from the image to be identified using the backbone network, and obtain the image features of the image to be identified. The depth prediction unit is used to predict the depth of the image to be identified based on the image features of the image to be identified using the depth estimation network, and obtain depth information. The depth estimation network is a neural network model obtained by jointly training sample images with semantic prediction network, parameter prediction network and mask prediction network.

[0295] In the above embodiments of this application, the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0296] In the above embodiments of this application, the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0297] In the above embodiments of this application, the three-dimensional reconstruction module includes: a first determining unit, a second determining unit, a third determining unit, and a generating unit.

[0298] The first determining unit is used to determine the target depth information corresponding to the target surface in the depth information; the second determining unit is used to determine the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; the third determining unit is used to determine the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and the generating unit is used to generate a planar point cloud based on the projection coordinates.

[0299] Example 9

[0300] Embodiments of this application may provide an image processing system, which may include an AR / VR device, a server, and a client. The AR / VR device may be any one of a group of AR / VR devices. Optionally, the image processing system includes: a processor; and a memory connected to the processor, used to provide the processor with instructions to process the following steps: acquiring an image to be identified, wherein the image to be identified contains the surface of at least one entity object; performing target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; and performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object; wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object.

[0301] Optionally, the processor may also execute the following steps: performing semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; performing parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; and performing mask prediction on the image features of the surface image using a mask prediction network to determine whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein the semantic prediction network, the parameter prediction network, and the mask prediction network are neural network models executed in parallel.

[0302] Optionally, the processor may further execute instructions for the following steps: a first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object; the category loss value is determined based on the predicted category and the true category of at least one sample object; the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object; a second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object; a third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object. Optionally, the processor may further execute instructions for the following steps: the semantic prediction network consists of two fully connected layers, which are respectively used to output the category of at least one entity object and the region of the surface of at least one entity object in the image to be recognized.

[0303] Optionally, the processor may also execute instructions for the following steps: the parameter prediction network consists of fully connected layers.

[0304] Optionally, the processor may also execute instructions for the following steps: the mask prediction network consists of convolutional layers.

[0305] Optionally, the processor may also execute the following steps: extracting image features from the image to be recognized using a backbone network to obtain image features of the image to be recognized, wherein the backbone network is a pre-trained neural network model; and cropping and deforming the image features of the image to be recognized based on the surface image to obtain image features of the surface image.

[0306] Optionally, the processor may also execute the following steps: extracting image features from the image to be recognized using a backbone network to obtain image features of the image to be recognized; performing target detection on the surface of at least one entity object in the image to be recognized using a candidate region network based on the image features of the image to be recognized to obtain detection results of the surface of at least one entity object, wherein the detection results include: at least one bounding box and the confidence of at least one bounding box, wherein the at least one bounding box is used to characterize the position of the surface of at least one entity object in the image to be recognized, and the confidence is used to characterize the accuracy of at least one bounding box, and the candidate region network is obtained by training a mask region convolutional neural network; determining a target bounding box from at least one bounding box based on the confidence of at least one bounding box, wherein the confidence of the target bounding box is greater than a first confidence threshold, and the confidence of other bounding boxes in at least one bounding box other than the target bounding box is less than or equal to the first confidence threshold; and cropping the image to be recognized based on the target bounding box to obtain a surface image.

[0307] Optionally, the processor may also execute instructions for the following steps: performing depth prediction on the image to be recognized to obtain depth information of the image to be recognized; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence level of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence levels of other semantic segmentation results corresponding to other surfaces of the at least one entity object besides the target surface are less than or equal to the second confidence threshold. Based on the target semantic segmentation result and the depth information, performing three-dimensional reconstruction of the target surface to obtain a planar point cloud corresponding to the target surface.

[0308] Optionally, the processor may also execute the following steps: extracting image features from the image to be recognized using a backbone network to obtain image features of the image to be recognized; and predicting the depth of the image to be recognized based on the image features of the image to be recognized using a depth estimation network to obtain depth information, wherein the depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network, and a mask prediction network.

[0309] Optionally, the processor may also execute instructions for the following steps: the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0310] Optionally, the processor may also execute instructions for the following steps: the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object, wherein the three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0311] Optionally, the processor may also execute instructions for the following steps: determining the target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projection coordinates.

[0312] Example 10

[0313] Embodiments of this application may provide an AR / VR device, which can be any AR / VR device from a group of AR / VR devices. Optionally, in this embodiment, the aforementioned AR / VR device may also be replaced by a terminal device such as a mobile terminal.

[0314] Optionally, in this embodiment, the AR / VR device described above may be located in at least one of a plurality of network devices in a computer network.

[0315] In this embodiment, the AR / VR device described above can execute the program code of the following steps in the image processing method: acquiring an image to be identified, wherein the image to be identified contains the surface of at least one entity object; performing target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object; wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object.

[0316] Optionally, the AR / VR device described above can execute program code that performs the following steps: using a semantic prediction network to perform semantic segmentation on the image features of the surface image to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; using a parameter prediction network to perform parameter prediction on the image features of the surface image to obtain the planar parameters of the surface of at least one entity object; using a mask prediction network to perform mask prediction on the image features of the surface image to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein, the semantic prediction network, the parameter prediction network, and the mask prediction network are neural network models executed in parallel.

[0317] Optionally, the AR / VR device described above can execute program code with the following steps: a first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object; the category loss value is determined based on the predicted category and the true category of at least one sample object; and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. A second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. A third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0318] Optionally, the AR / VR device described above can execute program code that performs the following steps: The semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be recognized.

[0319] Optionally, the AR / VR device described above can execute program code that includes the following steps: the parameter prediction network consists of fully connected layers.

[0320] Optionally, the AR / VR device described above can execute program code that includes the following steps: the mask prediction network consists of convolutional layers.

[0321] Optionally, the AR / VR device described above can execute program code for the following steps: extracting image features from the image to be recognized using a backbone network to obtain image features of the image to be recognized, wherein the backbone network is a pre-trained neural network model; and cropping and deforming the image features of the image to be recognized based on the surface image to obtain image features of the surface image.

[0322] Optionally, the AR / VR device described above can execute the following steps: using a backbone network to extract image features from the image to be recognized, obtaining image features of the image to be recognized; using a candidate region network to perform target detection on the surface of at least one entity object in the image to be recognized based on the image features of the image to be recognized, obtaining detection results for the surface of at least one entity object, wherein the detection results include: at least one bounding box and the confidence score of at least one bounding box, the at least one bounding box being used to characterize the position of the surface of at least one entity object in the image to be recognized, the confidence score being used to characterize the accuracy of at least one bounding box, and the candidate region network being obtained by training a mask region convolutional neural network; determining a target bounding box from the at least one bounding box based on the confidence score of the at least one bounding box, wherein the confidence score of the target bounding box is greater than a first confidence threshold, and the confidence scores of other bounding boxes in the at least one bounding box besides the target bounding box are less than or equal to the first confidence threshold; cropping the image to be recognized based on the target bounding box to obtain a surface image.

[0323] Optionally, the aforementioned AR / VR device can execute program code that performs the following steps: performing depth prediction on the image to be recognized to obtain depth information of the image; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence level of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence level of other semantic segmentation results corresponding to other surfaces of the at least one entity object besides the target surface is less than or equal to the second confidence threshold. Based on the target semantic segmentation result and depth information, performing three-dimensional reconstruction of the target surface to obtain a planar point cloud corresponding to the target surface.

[0324] Optionally, the AR / VR device described above can execute program code that performs the following steps: extracting image features from the image to be recognized using a backbone network to obtain image features of the image to be recognized; and using a depth estimation network to predict the depth of the image to be recognized based on the image features of the image to be recognized to obtain depth information. The depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network, and a mask prediction network.

[0325] Optionally, the AR / VR device described above can execute program code with the following steps: the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0326] Optionally, the AR / VR device described above can execute program code that performs the following steps: the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0327] Optionally, the AR / VR device described above can execute program code that performs the following steps: determining the target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projection coordinates.

[0328] Optionally, Figure 15 This is a structural block diagram of an AR / VR device according to an embodiment of this application. Figure 15 As shown, the AR / VR device A may include: one or more (only one is shown in the figure) processors 1502, memory 1504, storage controller 1505, and peripheral interface 1508, wherein the peripheral interface 1508 is connected to the radio frequency module 15010, the audio module 15012 and the display 15014.

[0329] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus in this application embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the AR / VR device A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0330] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring an image to be identified, wherein the image to be identified contains the surface of at least one entity object; performing target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object; wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object.

[0331] Optionally, the processor may also execute program code for the following steps: performing semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region where the surface of at least one entity object is located in the image to be identified; performing parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; and performing mask prediction on the image features of the surface image using a mask prediction network to determine whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein the semantic prediction network, the parameter prediction network, and the mask prediction network are neural network models executed in parallel.

[0332] Optionally, the processor may also execute program code for the following steps: a first loss value for the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object; the category loss value is determined based on the predicted category and the true category of at least one sample object; and the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object. A second loss value for the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object. A third loss value for the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0333] Optionally, the processor may also execute program code that performs the following steps: the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be recognized.

[0334] Optionally, the processor may also execute program code that includes the following steps: the parameter prediction network consists of fully connected layers.

[0335] Optionally, the processor may also execute program code that includes the following steps: the mask prediction network consists of convolutional layers.

[0336] Optionally, the processor may also execute program code for the following steps: extracting image features from the image to be identified using a backbone network to obtain image features of the image to be identified, wherein the backbone network is a pre-trained neural network model; and cropping and deforming the image features of the image to be identified based on the surface image to obtain image features of the surface image.

[0337] Optionally, the processor may also execute program code for the following steps: extracting image features from the image to be recognized using a backbone network to obtain image features of the image to be recognized; performing target detection on the surface of at least one entity object in the image to be recognized using a candidate region network based on the image features of the image to be recognized to obtain detection results of the surface of at least one entity object, wherein the detection results include: at least one bounding box and the confidence of at least one bounding box, wherein the at least one bounding box is used to characterize the position of the surface of at least one entity object in the image to be recognized, and the confidence is used to characterize the accuracy of at least one bounding box, and the candidate region network is obtained by training a mask region convolutional neural network; determining a target bounding box from at least one bounding box based on the confidence of at least one bounding box, wherein the confidence of the target bounding box is greater than a first confidence threshold, and the confidence of other bounding boxes in at least one bounding box other than the target bounding box is less than or equal to the first confidence threshold; cropping the image to be recognized based on the target bounding box to obtain a surface image.

[0338] Optionally, the processor may also execute program code for the following steps: performing depth prediction on the image to be recognized to obtain depth information of the image to be recognized; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence level of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence level of other semantic segmentation results corresponding to other surfaces of the at least one entity object besides the target surface is less than or equal to the second confidence threshold. Based on the target semantic segmentation result and depth information, performing three-dimensional reconstruction of the target surface to obtain a planar point cloud corresponding to the target surface.

[0339] Optionally, the processor may also execute program code for the following steps: extracting image features from the image to be identified using a backbone network to obtain image features of the image to be identified; and using a depth estimation network to predict the depth of the image to be identified based on the image features of the image to be identified to obtain depth information. The depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network, and a mask prediction network.

[0340] Optionally, the processor may also execute program code that performs the following steps: the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0341] Optionally, the processor may also execute program code for the following steps: the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object, wherein the three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and the camera intrinsic parameters.

[0342] Optionally, the processor may also execute program code for the following steps: determining the target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projection coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projection coordinates.

[0343] This application provides an image processing solution. By acquiring an image to be identified, wherein the image to be identified contains the surface of at least one entity object; performing target detection on the surface of the at least one entity object in the image to be identified to obtain a surface image of the surface of the at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; and performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of the at least one entity object. This achieves the technical effect of simultaneously obtaining the speech information of the image during image processing. It is noteworthy that this application, when processing images, not only can it extract image features of the surface image based on the surface image to obtain the image features of the image to be identified, but it can also perform semantic segmentation on the image to be identified based on the image features of the surface image, achieving the goal of obtaining a semantic segmentation result of the surface image. This achieves the technical effect of simultaneously obtaining the image features and semantic segmentation result of the surface image, thereby solving the technical problem in related technologies where plane estimation algorithms cannot obtain the semantic information of the plane, resulting in low accuracy of image segmentation.

[0344] Those skilled in the art will understand that Figure 15 The structure shown is for illustrative purposes only. AR / VR devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PCs, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 15This does not limit the structure of the aforementioned electronic device. For example, AR / VR device A may also include components that are more... Figure 15 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 15 The different configurations shown.

[0345] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0346] Example 11

[0347] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the image processing method provided in Embodiment 1.

[0348] Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in the AR / VR device terminal group in the AR / VR device network, or in any mobile terminal in the mobile terminal group.

[0349] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring an image to be identified, wherein the image to be identified contains the surface of at least one entity object; performing target detection on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object; performing image feature extraction on the surface image to obtain image features of the surface image; performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain a semantic segmentation result of the surface of at least one entity object; wherein the semantic segmentation result includes: the category of at least one entity object, the region in which the surface of at least one entity object is located in the image to be identified, the planar parameters of the surface of at least one entity object, and whether each pixel in the image to be identified belongs to the surface of at least one entity object.

[0350] Optionally, the computer-readable storage medium may also execute program code for the following steps: performing semantic segmentation on the image features of the surface image using a semantic prediction network to obtain the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be identified; performing parameter prediction on the image features of the surface image using a parameter prediction network to obtain the planar parameters of the surface of at least one entity object; and performing mask prediction on the image features of the surface image using a mask prediction network to obtain whether each pixel in the image to be identified belongs to the surface of at least one entity object; wherein the semantic prediction network, the parameter prediction network, and the mask prediction network are neural network models executed in parallel.

[0351] Optionally, the computer-readable storage medium may also execute program code for the following steps: a first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of at least one sample object; the category loss value is determined based on the predicted category and the true category of at least one sample object; the region loss value is determined based on the predicted region of the surface of at least one sample object and the true region of the surface of at least one sample object; a second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of at least one sample object; and a third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of at least one sample object.

[0352] Optionally, the computer-readable storage medium may also execute program code for the following steps: the semantic prediction network consists of two fully connected layers, which are used to output the category of at least one entity object and the region in which the surface of at least one entity object is located in the image to be recognized.

[0353] Optionally, the computer-readable storage medium may also execute program code for the following steps: the parameter prediction network consists of fully connected layers.

[0354] Optionally, the computer-readable storage medium may also execute program code for the following steps: the mask prediction network consists of convolutional layers.

[0355] Optionally, the computer-readable storage medium may also execute program code for the following steps: performing image feature extraction on the image to be identified using a backbone network to obtain image features of the image to be identified, wherein the backbone network is a pre-trained neural network model; and performing cropping and deformation operations on the image features of the image to be identified based on the surface image to obtain image features of the surface image.

[0356] Optionally, the computer-readable storage medium may also execute program code for the following steps: extracting image features from the image to be identified using a backbone network to obtain image features of the image to be identified; performing target detection on the surface of at least one entity object in the image to be identified using a candidate region network based on the image features of the image to be identified to obtain detection results of the surface of at least one entity object, wherein the detection results include: at least one bounding box and the confidence of at least one bounding box, the at least one bounding box being used to characterize the position of the surface of at least one entity object in the image to be identified, the confidence being used to characterize the accuracy of at least one bounding box, and the candidate region network being obtained by training a mask region convolutional neural network; determining a target bounding box from the at least one bounding box based on the confidence of the at least one bounding box, wherein the confidence of the target bounding box is greater than a first confidence threshold, and the confidence of other bounding boxes in the at least one bounding box besides the target bounding box is less than or equal to the first confidence threshold; and cropping the image to be identified based on the target bounding box to obtain a surface image.

[0357] Optionally, the computer-readable storage medium may also execute program code for the following steps: performing depth prediction on the image to be recognized to obtain depth information of the image to be recognized; determining a target surface from the surface of at least one entity object based on the semantic segmentation result, wherein the confidence level of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence level of other semantic segmentation results corresponding to other surfaces of the at least one entity object besides the target surface is less than or equal to the second confidence threshold. Based on the target semantic segmentation result and the depth information, performing three-dimensional reconstruction of the target surface to obtain a planar point cloud corresponding to the target surface.

[0358] Optionally, the computer-readable storage medium may also execute program code for the following steps: performing image feature extraction on the image to be identified using a backbone network to obtain image features of the image to be identified; and using a depth estimation network to predict the depth of the image to be identified based on the image features of the image to be identified to obtain depth information, wherein the depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network, and a mask prediction network.

[0359] Optionally, the computer-readable storage medium may also execute program code for the following steps: the depth estimation network consists of multiple first convolutional layers and second convolutional layers, wherein the activation functions of the first convolutional layers and the second convolutional layers are the same, and the convolutional kernels of the first convolutional layers and the second convolutional layers are different.

[0360] Optionally, the computer-readable storage medium may also execute program code for the following steps: the fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of at least one sample object, the three-dimensional coordinates being obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of at least one sample object, and camera intrinsic parameters.

[0361] Optionally, the computer-readable storage medium may also execute program code for the following steps: determining target depth information corresponding to the target surface in the depth information; determining the target three-dimensional coordinates of the target surface based on whether each pixel in the image to be identified belongs to the target surface and the target depth information; determining the projected coordinates of the target three-dimensional coordinates in the target plane based on the target plane parameters of the target surface; and generating a planar point cloud based on the projected coordinates.

[0362] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0363] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0364] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0365] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0366] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0367] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0368] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: Acquire an image to be identified, wherein the image to be identified contains the surface of at least one entity object; Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of the at least one entity object; Image feature extraction is performed on the surface image to obtain the image features of the surface image; Based on the image features of the surface image, semantic segmentation is performed on the image to be identified to obtain the semantic segmentation result of the surface of the at least one entity object. The semantic segmentation is used to represent processing by a parallel neural network model, which includes a semantic prediction network, a parameter prediction network, and a mask prediction network. The semantic segmentation result includes: the category of the at least one entity object, the region in which the surface of the at least one entity object is located in the image to be identified, the planar parameters of the surface of the at least one entity object, and whether each pixel in the image to be identified belongs to the surface of the at least one entity object.

2. The method according to claim 1, characterized in that, Based on the image features of the surface image, semantic segmentation is performed on the image to be identified to obtain the semantic segmentation result of the surface of the at least one entity object, including: The semantic prediction network is used to perform semantic segmentation on the image features of the surface image to obtain the category of the at least one entity object and the region where the surface of the at least one entity object is located in the image to be identified. The parameter prediction network is used to perform parameter prediction on the image features of the surface image to obtain the planar parameters of the surface of the at least one entity object; The mask prediction network is used to perform mask prediction on the image features of the surface image to determine whether each pixel in the image to be identified belongs to the surface of the at least one entity object.

3. The method according to claim 2, characterized in that, The first loss value corresponding to the semantic prediction network is determined based on the category loss value of at least one sample object contained in the sample image and the region loss value of the surface of the at least one sample object. The category loss value is determined based on the predicted category and the true category of the at least one sample object, and the region loss value is determined based on the predicted region of the surface of the at least one sample object and the true region of the surface of the at least one sample object. The second loss value corresponding to the parameter prediction network is determined based on the predicted plane parameters and the true plane parameters of the surface of the at least one sample object. The third loss value corresponding to the mask prediction network is determined based on the prediction result and the true result of whether each pixel in the sample image belongs to the surface of the at least one sample object.

4. The method according to claim 1, characterized in that, Image feature extraction is performed on the surface image to obtain the image features of the surface image, including: Image feature extraction is performed on the image to be identified using a backbone network to obtain the image features of the image to be identified, wherein the backbone network is a pre-trained neural network model; Based on the surface image, the image features of the image to be identified are cropped and deformed to obtain the image features of the surface image.

5. The method according to claim 1, characterized in that, Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of the at least one entity object, including: Image feature extraction is performed on the image to be identified using a backbone network to obtain the image features of the image to be identified; A candidate region network is used to perform target detection on the surface of at least one entity object in the image to be identified based on the image features of the image to be identified, and the detection result of the surface of the at least one entity object is obtained. The detection result includes at least one bounding box and the confidence score of the at least one bounding box. The at least one bounding box is used to characterize the position of the surface of the at least one entity object in the image to be identified, and the confidence score is used to characterize the accuracy of the at least one bounding box. The candidate region network is obtained by training a mask region convolutional neural network. A target bounding box is determined from the at least one bounding box based on the confidence level of the at least one bounding box, wherein the confidence level of the target bounding box is greater than a first confidence threshold, and the confidence levels of the other bounding boxes in the at least one bounding box besides the target bounding box are less than or equal to the first confidence threshold. The surface image is obtained by cropping the image to be identified based on the target bounding box.

6. The method according to claim 1, characterized in that, After performing semantic segmentation on the image to be identified based on the image features of the surface image to obtain the semantic segmentation result of the surface of the at least one entity object, the method further includes: Depth prediction is performed on the image to be identified to obtain the depth information of the image to be identified; Based on the semantic segmentation result, a target surface is determined from the surface of the at least one entity object, wherein the confidence of the target semantic segmentation result of the target surface is greater than a second confidence threshold, and the confidence of other semantic segmentation results corresponding to other surfaces in the surface of the at least one entity object other than the target surface is less than or equal to the second confidence threshold. Based on the target semantic segmentation result and the depth information, the target surface is reconstructed in three dimensions to obtain the planar point cloud corresponding to the target surface.

7. The method according to claim 6, characterized in that, Depth prediction is performed on the image to be identified to obtain its depth information, including: Image feature extraction is performed on the image to be identified using a backbone network to obtain the image features of the image to be identified; The depth information is obtained by using a depth estimation network to predict the depth of the image to be identified based on the image features of the image to be identified. The depth estimation network is a neural network model obtained by jointly training sample images with a semantic prediction network, a parameter prediction network and a mask prediction network.

8. The method according to claim 7, characterized in that, The fourth loss value corresponding to the depth estimation network is determined based on the true plane parameters of the surface of at least one sample object contained in the sample image and the three-dimensional coordinates of each pixel point within the surface of the at least one sample object. The three-dimensional coordinates are obtained based on the sample depth information of the sample image, the two-dimensional coordinates of each pixel point within the surface of the at least one sample object, and camera intrinsic parameters.

9. The method according to claim 6, characterized in that, Based on the target semantic segmentation result and the depth information, a three-dimensional reconstruction of the target surface is performed to obtain a planar point cloud corresponding to the target surface, including: Determine the target depth information corresponding to the target surface in the depth information; Based on whether each pixel in the image to be identified belongs to the target surface, and the target depth information, the target three-dimensional coordinates of the target surface are determined; Based on the target plane parameters of the target surface, determine the projected coordinates of the target's three-dimensional coordinates within the target plane; The planar point cloud is generated based on the projected coordinates.

10. An image processing method, characterized in that, include: In response to an input command applied to the operation interface, an image to be identified is displayed on the operation interface, wherein the image to be identified contains the surface of at least one entity object; In response to an instance segmentation command applied to the operation interface, the semantic segmentation result of the surface of the at least one entity object is displayed on the operation interface. The semantic segmentation result is obtained by semantically segmenting the image to be identified based on the image features of the surface image of the surface of the at least one entity object. The semantic segmentation is represented by processing through a parallel-executed neural network model, which includes a semantic prediction network, a parameter prediction network, and a mask prediction network. The semantic segmentation result includes: the category of the at least one entity object, the region where the surface of the at least one entity object is located in the image to be identified, the planar parameters of the surface of the at least one entity object, and whether each pixel in the image to be identified belongs to the surface of the at least one entity object. The surface image is obtained by performing target detection on the surface of the at least one entity object in the image to be identified, and the image features of the surface image are obtained by performing image feature extraction on the image to be identified based on the surface image.

11. An image processing method, characterized in that, include: An image to be identified is displayed on the screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image to be identified contains the surface of at least one entity object; Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of at least one entity object. Image feature extraction is performed on the surface image to obtain the image features of the surface image; Based on the image features of the surface image, semantic segmentation is performed on the image to be identified to obtain the semantic segmentation result of the surface of the at least one entity object. The semantic segmentation is used to represent processing by a parallel neural network model, which includes a semantic prediction network, a parameter prediction network, and a mask prediction network. The semantic segmentation result includes: the category of the at least one entity object, the region where the surface of the at least one entity object is located in the image to be identified, the planar parameters of the surface of the at least one entity object, and whether each pixel in the image to be identified belongs to the surface of the at least one entity object. The VR device or AR device is driven to display the semantic segmentation results of the surface of at least one entity object.

12. An image processing method, characterized in that, include: The image to be identified is obtained by calling a first interface, wherein the first interface includes a first parameter, the value of which is the image to be identified, and the image to be identified contains the surface of at least one entity object. Target detection is performed on the surface of at least one entity object in the image to be identified to obtain a surface image of the surface of the at least one entity object; Image feature extraction is performed on the surface image to obtain the image features of the surface image; Based on the image features of the surface image, semantic segmentation is performed on the image to be identified to obtain the semantic segmentation result of the surface of the at least one entity object. The semantic segmentation is used to represent processing by a parallel neural network model, which includes a semantic prediction network, a parameter prediction network, and a mask prediction network. The semantic segmentation result includes: the category of the at least one entity object, the region where the surface of the at least one entity object is located in the image to be identified, the planar parameters of the surface of the at least one entity object, and whether each pixel in the image to be identified belongs to the surface of the at least one entity object. The semantic segmentation result of the surface of the at least one entity object is output by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the semantic segmentation result of the surface of the at least one entity object.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 12.

14. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the method according to any one of claims 1 to 12.

Images