An image processing method, device, equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose an image processing method, device and equipment and a computer readable storage medium. The method comprises: obtaining a multi-view image set of a target object; predicting a three-dimensional coordinate point of the target object at a viewpoint corresponding to each view image in the multi-view image set based on position information of the target object in each view image; calibrating the three-dimensional coordinate point of the target object at each viewpoint based on a position relationship between the viewpoint corresponding to each view image and a calibration point, to obtain a three-dimensional calibration coordinate point of the target object based on the calibration point; and fusing the three-dimensional calibration coordinate point of the target object based on the calibration point, to obtain a three-dimensional model of the target object. The three-dimensional calibration coordinate point of the target object based on the calibration point can be obtained based on the position information of the target object in the multi-view image, and the three-dimensional model of the target object is shaped by fusing the three-dimensional calibration coordinate point based on the calibration point, thereby improving the modeling efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an image processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the continuous development of computer technology, Augmented Reality (AR) technology has been widely applied in various aspects of life; for example, AR games, AR home viewing, AR home furnishing, and so on. AR technology relies on 3D models; taking AR home furnishing as an example, to display target furniture using AR technology, a 3D model of the target furniture needs to be built first. Practice has shown that most 3D models are created by 3D modelers based on relevant data of the actual object (such as length, width, and height), resulting in relatively low modeling efficiency. Summary of the Invention

[0003] The present invention provides an image processing method, apparatus, device, and computer-readable storage medium, which can significantly improve the efficiency of modeling.

[0004] On one hand, embodiments of this application provide an image processing method, including:

[0005] Obtain a multi-viewpoint image set of the target object, which contains multiple viewpoint images; these multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints.

[0006] Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, predict the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image;

[0007] Based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, the three-dimensional coordinate points of the target object under each viewpoint are calibrated to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0008] The three-dimensional calibration coordinates of the target object based on the calibration points are fused to obtain the three-dimensional model of the target object.

[0009] On one hand, embodiments of this application provide an image processing apparatus, which includes:

[0010] The acquisition unit is used to acquire a multi-viewpoint image set of the target object, which contains multiple viewpoint images; the multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints respectively;

[0011] The processing unit is used to predict the three-dimensional coordinates of the target object in each viewpoint image based on the position information of the target object in each viewpoint image of the multi-viewpoint image set.

[0012] And it is used to calibrate the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, so as to obtain the three-dimensional calibration coordinates of the target object based on the calibration point;

[0013] And it is used to fuse the three-dimensional calibration coordinates of the target object based on the calibration points to obtain the three-dimensional model of the target object.

[0014] In one implementation, the processing unit is configured to predict the three-dimensional coordinates of the target object at the viewpoint corresponding to each viewpoint image, based on the position information of the target object in each viewpoint image, specifically configured to:

[0015] The position information of the target object in each viewpoint image of the multi-viewpoint image set is processed by convolution to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0016] In one implementation, the processing unit is configured to perform convolution processing on the position information of the target object in each viewpoint image of the multi-viewpoint image set to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image, specifically configured to:

[0017] Each viewpoint image is encoded to obtain a binary mask for each viewpoint image;

[0018] The binary mask of each viewpoint image is generated by calling the 3D structure generation model to obtain the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0019] The 3D structure generation model includes a 2D convolutional layer, which is used to perform 2D convolution operations on the binary masks of each viewpoint image.

[0020] In one embodiment, the processing unit is further configured to:

[0021] Obtain the spatial location of the reference viewpoint, which refers to any viewpoint other than the viewpoint corresponding to each viewpoint image in the multi-viewpoint image set;

[0022] Based on the spatial position of the reference viewpoint, the 3D model is projected to obtain the projected image of the 3D model of the target object under the reference viewpoint.

[0023] The projected image of the 3D model at the reference viewpoint is processed by pseudo-rendering to obtain the predicted image of the 3D model at the reference viewpoint.

[0024] Based on the loss value between the predicted image and the labeled image at the reference viewpoint, the parameters in the 3D structure generation model are optimized to obtain the optimized 3D structure generation model.

[0025] In one implementation, the processing unit is configured to perform pseudo-rendering processing on the projected image of the 3D model from a reference viewpoint to obtain a predicted image of the 3D model from the reference viewpoint, specifically for:

[0026] The projection image of the 3D model at the reference viewpoint is upsampled by a factor of U to obtain the upsampled image, where U is a positive integer.

[0027] The updated upsampled image is downsampled, and the pixel with the smallest depth value among all pixel positions in the downsampled image is retained to obtain the predicted image of the 3D model at the reference viewpoint.

[0028] In one implementation, the processing unit is configured to optimize the 3D structure generation model based on the loss value between the predicted image and the labeled image at the reference viewpoint, specifically by:

[0029] The depth loss component is calculated based on the difference between the depth image corresponding to the predicted image under the reference viewpoint and the depth image corresponding to the labeled image under the reference viewpoint.

[0030] The mask loss component is calculated based on the difference between the mask image corresponding to the predicted image under the reference viewpoint and the mask image corresponding to the labeled image under the reference viewpoint.

[0031] By using the depth loss component and the mask loss component, the loss value between the predicted image and the labeled image at the reference viewpoint is obtained.

[0032] The parameters in the 3D structure generation model are optimized based on the loss value so that the 3D structure generation model meets the optimization conditions.

[0033] In one implementation, the processing unit is configured to calibrate the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, thereby obtaining the three-dimensional calibration coordinates of the target object based on the calibration point. Specifically, it is configured to:

[0034] Based on the spatial position of the viewpoint and the spatial position of the calibration point corresponding to each viewpoint image, determine the calibration matrix corresponding to each viewpoint;

[0035] By using the calibration matrix corresponding to each viewpoint, the three-dimensional coordinate points under that viewpoint are calibrated to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0036] In one embodiment, the processing unit is further configured to:

[0037] Construct a 3D point cloud map and obtain the placement information of the 3D model of the target object in the 3D point cloud map;

[0038] In response to the confirmation of the placement location information, the 3D model of the target object is added to the location indicated by the placement location information in the 3D point cloud map.

[0039] In one implementation, the processing unit is used to construct a three-dimensional point cloud map, specifically for:

[0040] Obtain the position information of the reference object;

[0041] Based on the position information of the reference object, a reference coordinate system is established. The reference coordinate system is associated with projection information, which is used to indicate the projection relationship between the reference coordinate system and the real world.

[0042] A 3D point cloud map is constructed in a reference coordinate system using projection information.

[0043] In one embodiment, the processing unit is further configured to:

[0044] In response to viewing operations on a 3D point cloud map, obtain the observation position of the 3D point cloud map;

[0045] Based on the observation location, a planar projection is performed on the 3D point cloud map to obtain the observation effect image of the 3D point cloud map at the observation location.

[0046] Accordingly, this application provides an image processing apparatus, which includes:

[0047] A processor is used to load and execute computer programs;

[0048] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image processing method.

[0049] Accordingly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described image processing method.

[0050] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned image processing method.

[0051] In this embodiment, a multi-viewpoint image set of the target object is obtained. Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image are predicted. According to the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, the 3D coordinates of the target object under each viewpoint are calibrated to obtain the 3D calibration coordinates of the target object based on the calibration points. The 3D calibration coordinates of the target object based on the calibration points are fused to obtain the 3D model of the target object. It can be seen that the 3D calibration coordinates of the target object based on the position information of the target object in the multi-viewpoint images can be obtained. By fusing the 3D calibration coordinates based on the calibration points to shape the 3D model of the target object, the modeling efficiency is significantly improved. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A scene architecture diagram of an image processing system provided in this application embodiment;

[0054] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0055] Figure 3 A schematic flowchart illustrating another image processing method provided in an embodiment of this application;

[0056] Figure 4 A schematic diagram of model optimization provided for an embodiment of this application;

[0057] Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0058] Figure 6 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] Artificial Intelligence (AI): AI refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. This application's embodiments mainly involve, during image processing, using a 3D structure generation model to convolve the binary masks of the target object's images from various viewpoints to obtain the target object's 3D coordinates at the corresponding viewpoints in each viewpoint image.

[0061] AI technology is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large application processing technologies, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0062] Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of AI and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning / deep learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning. This application's embodiments mainly involve training a 3D structure generation model by analyzing the difference between the labeled image at a reference viewpoint and the predicted image of the point cloud model at the reference viewpoint, to obtain an optimized 3D structure generation model.

[0063] Deep learning: The concept of deep learning originates from research on artificial neural networks. A multilayer perceptron with multiple hidden layers is a type of deep learning structure. Deep learning discovers distributed feature representations of data by combining low-level features to form more abstract high-level representations of attribute categories or features. This application's embodiments mainly involve encoding images of a target object from various viewpoints using an encoder to obtain binary masks for each viewpoint image.

[0064] This application provides an image processing method and system to improve modeling efficiency. Please refer to... Figure 1 , Figure 1 This is a scene architecture diagram of an image processing system provided in an embodiment of this application. Figure 1 As shown, the image processing system may include a terminal device 101 and a server 102. The image processing method provided in this embodiment can be executed by the server 102. The terminal device 101 may include, but is not limited to, devices with display functions such as smartphones (e.g., Android phones, iOS phones), tablets, portable personal computers, mobile internet devices (MIDs), and vehicle terminals. This embodiment does not limit the types of devices in this application. The server 102 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This embodiment does not limit the types of cloud servers in this application.

[0065] It should be noted that, Figure 1 The terminal and the server can be connected directly or indirectly through wired or wireless communication, and this application does not impose any limitations on this. The number of terminal devices and servers is only for illustrative purposes and does not constitute an actual limitation of this application; for example, the image processing system may also include terminal device 103, or server 104, etc.

[0066] In practical implementation, the general principle of image processing methods is as follows:

[0067] (1) Server 102 obtains a multi-viewpoint image set of the target object sent by terminal device 101. The multi-viewpoint image set contains multiple viewpoint images. The multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints. For example, suppose the multi-viewpoint image set includes viewpoint image 1 to viewpoint image 3. Viewpoint image 1 is obtained by taking pictures of the target object from a first viewpoint, and viewpoint images 2 and 3 are obtained by taking pictures of the target object from a second viewpoint. The first viewpoint and the second viewpoint are two different viewpoints. The target object can be furniture, decorations, cars, workpieces, animals, etc., and this application does not limit this.

[0068] (2) Server 102 predicts the three-dimensional coordinates of the target object at the viewpoint corresponding to each viewpoint image based on the position information of the target object in each viewpoint image of the multi-view image set. The position information of the target object in each viewpoint image of the multi-view image set is used to indicate the position of the target object in each viewpoint image; for example, the position information can be coordinate points used to indicate the position of the target object in each viewpoint image, or it can be pixels in each viewpoint image used to display the target object. Server 102 performs convolution processing on the position information of the target object in each viewpoint image of the multi-view image set to obtain the three-dimensional coordinates of the target object at the viewpoint corresponding to each viewpoint image; wherein, the convolution processing can specifically refer to two-dimensional convolution processing or three-dimensional convolution processing. In one embodiment, server 102 can perform convolution processing on the position information of the target object in each viewpoint image of the multi-view image set using a neural network to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image. The neural network includes a two-dimensional convolutional layer or a three-dimensional convolutional layer. The two-dimensional convolutional layer is used to perform two-dimensional convolution processing on the position information of the target object in each viewpoint image of the multi-view image set, and the three-dimensional convolutional layer is used to perform three-dimensional convolution processing on the position information of the target object in each viewpoint image of the multi-view image set.

[0069] (3) Server 102 calibrates the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, thereby obtaining the three-dimensional calibration coordinates of the target object based on the calibration point. The calibration point can be any point among the viewpoints corresponding to each viewpoint image, or a point outside the viewpoints corresponding to each viewpoint image; the calibration point can be preset or specified by the user. The position of the viewpoint corresponding to each viewpoint image refers to the spatial position of the target object when the image acquisition device captures it; for example, assuming viewpoint image 1 is obtained by the image acquisition device capturing the target object at spatial position A, then the spatial position of the viewpoint corresponding to viewpoint image 1 is spatial position A. The three-dimensional calibration coordinates of the target object under each viewpoint refer to the absolute positioning coordinates of the three-dimensional coordinates of the target object under each viewpoint relative to the calibration point. For example, the three-dimensional coordinates of point A in the target object under viewpoint 1 are (a, b, c). After calibrating the three-dimensional coordinates of point A under viewpoint 1 based on the positional relationship between the calibration point and viewpoint 1, the three-dimensional calibration coordinates of point A in the target object based on the calibration point are obtained as (e, f, g).

[0070] (4) Server 102 fuses the three-dimensional calibration coordinates of the target object based on the calibration points to obtain a three-dimensional model of the target object. Fusion refers to combining the three-dimensional calibration coordinates of the target object based on the calibration points to obtain a three-dimensional model of the target object. After obtaining the three-dimensional model of the target object, server 102 returns the three-dimensional model of the target object to terminal device 101.

[0071] In practical applications, the computer program corresponding to the image processing method provided in this application can be mounted on the aforementioned terminal device 101 or server 102. It is understood that when the computer program corresponding to the image processing method provided in this application is mounted on the terminal device 101, the image processing method can be executed independently by the terminal device 101.

[0072] In this embodiment, a multi-viewpoint image set of the target object is obtained. Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image are predicted. According to the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, the 3D coordinates of the target object under each viewpoint are calibrated to obtain the 3D calibration coordinates of the target object based on the calibration points. The 3D calibration coordinates of the target object based on the calibration points are fused to obtain the 3D model of the target object. It can be seen that the 3D calibration coordinates of the target object based on the position information of the target object in the multi-viewpoint images can be obtained. By fusing the 3D calibration coordinates based on the calibration points to shape the 3D model of the target object, the modeling efficiency is significantly improved.

[0073] The image processing method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0074] Please see Figure 2 , Figure 2 This is a flowchart illustrating an image processing method provided in an embodiment of this application. The image processing method can be executed by an image processing device, which may specifically be... Figure 1 The terminal device 101 or server 102 in the method; the method includes steps S201-S204, wherein:

[0075] S201. Obtain a multi-viewpoint image set of the target object.

[0076] A multi-viewpoint image set contains multiple viewpoint images; these multiple viewpoint images are obtained by capturing the target object from multiple viewpoints; the multi-viewpoint image set of the target object may include one or more images of the target object from a single viewpoint; for example, suppose the multi-viewpoint image set includes viewpoint images 1-3, where viewpoint image 1 was captured from a first viewpoint, and viewpoint images 2 and 3 were captured from a second viewpoint, with the first and second viewpoints being two different viewpoints. The target object may specifically be furniture, decorations, automobiles, workpieces, animals, etc., and this application does not impose any limitations on this.

[0077] In one embodiment, a multi-viewpoint image set of the target object can be obtained by taking pictures of the target object from different positions using an image acquisition device. The image acquisition device can be mounted on a standalone image acquisition device or on an image processing device.

[0078] S202. Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, predict the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0079] The position information of the target object in each viewpoint image of the multi-viewpoint image set is used to indicate the position of the target object in each viewpoint image; the position information of the target object in any viewpoint image includes: the pixel points corresponding to the visible area of ​​the target object in that viewpoint image.

[0080] The image processing device performs convolution processing on the position information of the target object in each viewpoint image of the multi-viewpoint image set to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image; wherein, the convolution processing can specifically refer to two-dimensional convolution processing or three-dimensional convolution processing.

[0081] In one embodiment, the image processing device performs convolution processing on the position information of the target object in each viewpoint image of the multi-view image set using a neural network to obtain the three-dimensional coordinates of the target object at the viewpoint corresponding to each viewpoint image. The neural network includes a two-dimensional convolutional layer or a three-dimensional convolutional layer. The two-dimensional convolutional layer is used to perform two-dimensional convolution processing on the position information of the target object in each viewpoint image of the multi-view image set, and the three-dimensional convolutional layer is used to perform three-dimensional convolution processing on the position information of the target object in each viewpoint image of the multi-view image set.

[0082] In one embodiment, the neural network includes an encoder and a 3D structure generation model. The neural network performs convolution processing on the position information of the target object in each viewpoint image of the multi-viewpoint image set to obtain the 3D coordinates of the target object at each viewpoint image. This involves encoding each viewpoint image using the encoder to obtain a binary mask for each viewpoint image. The binary mask of each viewpoint image carries the feature information of the pixels corresponding to the visible area of ​​the target object in that viewpoint image. After obtaining the binary mask of each viewpoint image, the 3D structure generation model is invoked to generate the 3D coordinates of the target object at each viewpoint image. The 3D structure generation model includes a 2D convolutional layer, which performs a 2D convolution operation on the binary mask of each viewpoint image. The 3D coordinates of the target object at each viewpoint image are predicted based on the result of the 2D convolution operation on the binary mask of that viewpoint image.

[0083] Optionally, the 3D structure generation model includes 3D convolutional layers, which perform 3D convolution operations on the binary mask of each viewpoint image. In this case, the 3D coordinates of the target object at the viewpoint corresponding to each viewpoint image are predicted based on the 3D convolution operation result of the binary mask of that viewpoint image. Practice has shown that using 2D convolutional layers to perform convolution processing on the position information of the target object in each viewpoint image of a multi-viewpoint image set is more computationally efficient than using 3D convolutional layers.

[0084] S203. Based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, calibrate the three-dimensional coordinate points of the target object under each viewpoint to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0085] The calibration point can be any point within the viewpoint corresponding to each viewpoint image, or it can be a point outside the viewpoint corresponding to each viewpoint image; the calibration point can be preset or specified by the user.

[0086] The image processing device determines the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image based on the spatial position of the viewpoint and the calibration point corresponding to each viewpoint image. The spatial position of the viewpoint corresponding to each viewpoint image refers to the spatial position where the image acquisition device is located when it takes a picture of the target object. For example, assuming that viewpoint image 1 is obtained by the image acquisition device taking a picture of the target object at spatial position A, then the spatial position of the viewpoint corresponding to viewpoint image 1 is spatial position A.

[0087] The image processing device calibrates the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, thereby obtaining the three-dimensional calibration coordinates of the target object based on the calibration point. For example, let the positional relationship between viewpoint A corresponding to viewpoint image 1 and the calibration point be the first positional relationship, and the positional relationship between three-dimensional coordinate point 1 under viewpoint A and viewpoint A be the second positional relationship. Based on the first positional relationship and the second positional relationship, the three-dimensional calibration coordinates of three-dimensional coordinate point 1 based on the calibration point can be obtained (i.e., the coordinates of three-dimensional coordinate point 1 based on the calibration point are determined).

[0088] S204. Fuse the three-dimensional calibration coordinates of the target object based on the calibration points to obtain the three-dimensional model of the target object.

[0089] In one implementation, the image processing device combines the three-dimensional calibration coordinates of the target object based on the calibration points to obtain a three-dimensional model of the target object.

[0090] In this embodiment, a multi-viewpoint image set of the target object is obtained. Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image are predicted. According to the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, the 3D coordinates of the target object under each viewpoint are calibrated to obtain the 3D calibration coordinates of the target object based on the calibration points. The 3D calibration coordinates of the target object based on the calibration points are fused to obtain the 3D model of the target object. It can be seen that the 3D calibration coordinates of the target object based on the position information of the target object in the multi-viewpoint images can be obtained. By fusing the 3D calibration coordinates based on the calibration points to shape the 3D model of the target object, the modeling efficiency is significantly improved.

[0091] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating another image processing method provided in an embodiment of this application. This image processing method can be executed by an image processing device, which may specifically be... Figure 1 The terminal device 101 or server 102 in the method; the method includes steps S301-S310, wherein:

[0092] S301. Obtain a multi-viewpoint image set of the target object.

[0093] For a detailed implementation of step S301, please refer to Figure 2 The implementation method of step S201 will not be described in detail here.

[0094] S302. Encode each viewpoint image to obtain the binary mask of each viewpoint image.

[0095] In one implementation, the image processing device calls an encoder to encode each viewpoint image to obtain a binary mask for each viewpoint image.

[0096] S303. Call the 3D structure generation model to generate the binary mask of each viewpoint image to obtain the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0097] The 3D structure generation model includes a 2D convolutional layer, which is used to perform 2D convolution operations on the binary mask of each viewpoint image. The 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image are predicted based on the 2D convolution operation result of the binary mask of that viewpoint image.

[0098] Optionally, the 3D structure generation model can also include 3D convolutional layers. These layers perform 3D convolution operations on the binary mask of each viewpoint image. The 3D coordinates of the target object at each viewpoint image are predicted based on the 3D convolution result of the binary mask of that viewpoint image. In practice, using 2D convolutional layers saves computational resources and improves computational efficiency compared to using 3D convolutional layers.

[0099] S304. Determine the calibration matrix corresponding to each viewpoint based on the spatial position of the viewpoint and the spatial position of the calibration point corresponding to each viewpoint image.

[0100] The calibration matrix is ​​used to estimate the relevant parameters of the viewpoint image and the image sensor. These parameters can be used to measure the size of a target object in the real world, or to indicate the positional relationship between the viewpoint and the calibration point in the viewpoint image.

[0101] S305. Using the calibration matrix corresponding to each viewpoint, calibrate the three-dimensional coordinate points under that viewpoint to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0102] In one implementation, the image processing device converts the three-dimensional coordinate points under each viewpoint into three-dimensional coordinate points under the calibration point (i.e., three-dimensional calibration coordinate points) using a calibration matrix corresponding to each viewpoint. Formula 1 is an expression provided in this application embodiment for calibrating the three-dimensional coordinate points under the viewpoint corresponding to the viewpoint image:

[0103]

[0104] in, Let the i-th point be the three-dimensional calibration coordinate point based on the calibration point. K represents the inverse of the orientation matrix for the nth viewpoint; -1 The matrix representing the inverse of the calibration matrix; Let i be the three-dimensional coordinates of the i-th point from the n-th viewpoint. It can be represented as Let i be the coordinates of the i-th point. t represents the depth value of the i-th point; n This is the offset vector for the nth viewpoint.

[0105] S306. The three-dimensional calibration coordinates of the target object based on the calibration points are fused to obtain the three-dimensional model of the target object.

[0106] In one implementation, after obtaining the 3D model of the target object, the parameters in the 3D structure generation model can be optimized based on the 3D model of the target object to obtain an optimized 3D structure generation model. Specific optimization steps include steps S11-S14:

[0107] S11. Obtain the spatial position of the reference viewpoint.

[0108] A reference viewpoint refers to any viewpoint other than the viewpoint corresponding to each viewpoint image in a multi-viewpoint image set. In one implementation, the spatial position of the reference viewpoint is determined by an orientation matrix and an offset vector. The orientation matrix and offset vector of the reference viewpoint are used to indicate the positional relationship between the reference viewpoint and the calibration point, and the spatial position of the reference viewpoint can be determined based on the positional relationship between the reference viewpoint and the calibration point.

[0109] S12. Based on the spatial position of the reference viewpoint, project the 3D model to obtain the projected image of the 3D model of the target object under the reference viewpoint.

[0110] The projection image of the target object's 3D model onto a reference viewpoint is used to simulate the viewpoint image of the target object from that viewpoint. Since the 3D model is projected onto a 2D plane, each pixel location in the projection image can include one or more pixels. Each pixel carries its position coordinates and depth value information. The depth value of each pixel indicates its distance from the reference viewpoint; the greater the distance from the reference viewpoint, the greater the depth value of that pixel.

[0111] Formula 2 is an expression provided in this application embodiment for projecting points on a three-dimensional model of a target object onto a two-dimensional plane corresponding to a reference viewpoint:

[0112]

[0113] Formula 2 is the inverse operation of Formula 1. Wherein, Let i be the three-dimensional coordinates of the i-th point from the k-th reference viewpoint. It can be represented as Let i be the coordinates of the i-th point. R is the depth value of the i-th point; K is the calibration matrix, which can be obtained from the positional relationship between the reference viewpoint and the calibration point; k Let t be the orientation matrix for the k-th reference viewpoint. k Let k be the offset vector of the k-th reference viewpoint. Let i be the three-dimensional calibration coordinate point based on the calibration point.

[0114] S13. Perform pseudo-rendering on the projected image of the 3D model at the reference viewpoint to obtain the predicted image of the 3D model at the reference viewpoint.

[0115] The predicted image is a pixelated depth image of the 3D model of the target object from a reference viewpoint. A pixelated depth image is an image in which the information of each pixel contains the depth value of the pixel. Formula 3 is an expression provided by an embodiment of this application for performing pseudo-rendering processing on the projected image of the 3D model from a reference viewpoint to obtain the predicted image of the 3D model from the reference viewpoint:

[0116]

[0117] in, A pixelated depth image of the 3D model of the target object from the reference viewpoint. The set of three-dimensional calibration coordinate points of the target object based on the calibration points. This refers to using a pseudo-renderer to perform pseudo-rendering on a set of three-dimensional calibration coordinate points based on calibration points for the target object.

[0118] Furthermore, the mask corresponding to the predicted image can be obtained based on the predicted image from the reference viewpoint using the 3D model. In this context, the mask for pixels belonging to the target object is 1, and the mask for pixels not belonging to the target object is 0.

[0119] In one embodiment, the projected image of the 3D model at the reference viewpoint is upsampled by a factor of U to obtain an upsampled image, where U is a positive integer. After obtaining the upsampled image, the depth values ​​of each pixel in the upsampled image are updated to the reciprocal of the original depth value; for example, if the depth value of pixel A before the update is 2, then the depth value of pixel A after the update is 0.5; as another example, if the depth value of pixel B before the update is 5, then the depth value of pixel B after the update is 0.2. After updating the depth values ​​of each pixel in the upsampled image, a convolutional kernel of size U is used to downsample the upsampled image (e.g., max pooling) to obtain the predicted image of the 3D model at the reference viewpoint.

[0120] It should be noted that by downsampling an upsampled image using a convolutional kernel of size U, the upsampled image can be restored to the resolution of the projected image. By updating the depth value of each pixel in the upsampled image to the reciprocal of that pixel's depth value and performing max pooling on the upsampled image, the pixel with the smallest depth value among all pixel positions in the downsampled image can be retained.

[0121] In practical applications, rendered images are often non-differentiable, making them unsuitable for direct use and integration into deep learning frameworks. Therefore, this application employs pseudo-rendering on the projected image of a 3D model from a reference viewpoint. Pseudo-rendering is differentiable rendering, a process that allows for differentiation and is divided into forward and inverse processes. The forward process is similar to traditional rendering, inputting the model and parameters to obtain an image. The inverse process involves calculating the derivative of pixels with respect to scene parameters. Differentiable rendering requires both processes, obtaining not only the rendered result but also the derivative of the rendered result with respect to the input. Differentiable rendering maintains the distinguishability and parallelizability of the predicted image of the 3D model from the reference viewpoint.

[0122] S14. Based on the loss value between the predicted image under the reference viewpoint and the labeled image under the reference viewpoint, optimize the parameters in the 3D structure generation model to obtain the optimized 3D structure generation model.

[0123] The depth loss component is calculated based on the difference between the depth image corresponding to the predicted image at the reference viewpoint and the depth image corresponding to the labeled image at the reference viewpoint. The labeled image at the reference viewpoint refers to the viewpoint image obtained by capturing the target object at the reference viewpoint. Formula 4 is a calculation expression for the depth loss component provided in an embodiment of this application:

[0124]

[0125] Among them, L depth This represents the depth loss component, where k represents the k-th reference viewpoint out of the K reference viewpoints. Z is the predicted image of the 3D model of the target object from the k-th reference viewpoint. k Let ||x|| be the labeled image of the target object at the k-th reference viewpoint, and ||x|| denote the norm of x.

[0126] The mask loss component is calculated based on the difference between the mask image corresponding to the predicted image under the reference viewpoint and the mask image corresponding to the labeled image under the reference viewpoint; Formula 5 is a calculation expression for the mask loss component provided in an embodiment of this application:

[0127]

[0128] Among them, L mask This represents the mask loss component, where k represents the k-th reference viewpoint out of K reference viewpoints, and M... k This is the mask for the annotated image of the target object at the k-th reference viewpoint. The mask for the predicted image of the 3D model of the target object at the k-th reference viewpoint.

[0129] By using the depth loss component and the mask loss component, the loss value between the predicted image and the labeled image at the reference viewpoint is obtained; Formula 6 is a calculation expression for the loss value provided in an embodiment of this application:

[0130] L = L mask +λ·L depth Formula 6

[0131] Where L represents the loss value between the predicted image at the reference viewpoint and the labeled image at the reference viewpoint, L mask L represents the mask loss component. depth Let λ represent the depth loss component, and λ be the weighting coefficient, with a value range of (0,1).

[0132] The parameters in the 3D structure generation model are optimized based on the loss value so that the 3D structure generation model meets the optimization conditions; for example, so that L obtained based on the optimized 3D structure generation model is less than the loss threshold.

[0133] Figure 4 This is a schematic diagram illustrating a model optimization provided in an embodiment of this application. Figure 4 As shown, firstly, the n viewpoint images of the target object are encoded using an encoder to obtain binary masks corresponding to the n viewpoint images; then, a 3D structure generation model is called to generate the binary mask corresponding to each viewpoint image, obtaining the 3D coordinates of the target object from the n viewpoints; based on the spatial positions of the n viewpoints (e.g., via (R1,t1)...(R... N ,t NLet be the spatial positions of n viewpoints and the positional relationship between the calibration points. The 3D coordinates of the n viewpoints are calibrated to obtain the 3D calibration coordinates of the target image based on the calibration points. Then, the 3D calibration coordinates of the target object based on the calibration points are fused to obtain the 3D model of the target object. Further, after obtaining the 3D model of the target object, the 3D model can be projected based on the spatial positions of k reference viewpoints to obtain k projected images of the target object's 3D model under the k reference viewpoints. Then, pseudo-rendering processing is performed on the k projected images to obtain the predicted images of the target object's 3D model under the k reference viewpoints. Next, the loss value is calculated based on the difference between the predicted image under each reference viewpoint and the labeled image under that reference point. One or more parameters in the 3D structure generation model are adjusted based on the loss value to make the 3D structure generation model meet optimization conditions; for example, making the loss value obtained based on the optimized 3D structure generation model less than a loss threshold.

[0134] It should be noted that the above steps S11-S14 can be performed by a separate model optimization device or by an image processing device, and this application does not limit this.

[0135] It is evident that optimizing the 3D structure generation model by using the loss value between the predicted image and the labeled image at the reference viewpoint improves the prediction accuracy of the 3D model of the target object. It should be noted that optimizing the 3D structure generation model using reference viewpoints other than those corresponding to the viewpoints of each viewpoint image in the multi-viewpoint image set can distribute the error evenly across the reference viewpoint, rather than concentrating it on the viewpoints corresponding to each viewpoint image in the multi-viewpoint image set.

[0136] S307. Construct a 3D point cloud map and obtain the placement information of the 3D model of the target object in the 3D point cloud map.

[0137] In one implementation, the image processing device acquires the position information of a reference object, which can be specified by a user; for example, the user designates a target object as the specified reference object and specifies the corresponding three-dimensional coordinates of the reference object; the position information of the reference object can also be pre-set. After acquiring the position information of the reference object, a reference coordinate system is established based on the position information of the reference object. The reference coordinate system is associated with projection information, which is used to indicate the projection relationship between the reference coordinate system and the real world; for example, the projection information can be scaling information.

[0138] Furthermore, based on projection information, the image processing device can construct a three-dimensional point cloud map in a reference coordinate system; for example, it can track and expand the three-dimensional point cloud map in the reference coordinate system using Simultaneous Localization and Mapping (SLAM) technology.

[0139] S308. In response to the confirmation of the placement position information, add the 3D model of the target object to the position indicated by the placement position information in the 3D point cloud map.

[0140] In one implementation, once the user determines the placement position of the 3D model of the target object in the point cloud map, the 3D model of the target object, after being projected according to the projection information, is added to the position indicated by the placement position information in the 3D point cloud map.

[0141] S309. In response to the viewing operation of the 3D point cloud map, obtain the observation position of the 3D point cloud map.

[0142] In one implementation, after a user selects an observation location in a 3D point cloud map, the image processing device obtains the coordinates of that observation location in the 3D point cloud map.

[0143] S310. Based on the observation position, perform planar projection on the three-dimensional point cloud map to obtain the observation effect image of the three-dimensional point cloud map at the observation position.

[0144] Taking AR home furnishing as an example, after a user places a furniture model in a virtual room, they can choose different viewpoints to observe the furniture model. Whenever the user selects an observation position, the furniture model in the virtual room is projected onto a plane based on that observation position to obtain a two-dimensional observation image of the furniture model in the virtual room at the observation position.

[0145] Optionally, users can save or share the observation images at various observation locations for later use; for example, after obtaining multiple observation images, they can compare the images.

[0146] In this embodiment, a multi-viewpoint image set of the target object is obtained. Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image are predicted. According to the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, the 3D coordinates of the target object under each viewpoint are calibrated to obtain the 3D calibration coordinates of the target object based on the calibration points. The 3D calibration coordinates of the target object based on the calibration points are fused to obtain the 3D model of the target object. It can be seen that the 3D calibration coordinates of the target object based on the position information of the target object in the multi-viewpoint images can be obtained. By fusing the 3D calibration coordinates based on the calibration points to shape the 3D model of the target object, the modeling efficiency is significantly improved. Furthermore, by employing two-dimensional convolutional layers to perform two-dimensional convolution operations on the binary masks of each viewpoint image, computational resources are saved and computational efficiency is improved. By performing pseudo-rendering processing on the projection image of the 3D model under the reference viewpoint, the distinguishability and parallelizability of the predicted image of the 3D model under the reference viewpoint are maintained. By optimizing the 3D structure generation model with the loss value between the predicted image under the reference viewpoint and the labeled image under the reference viewpoint, the prediction accuracy of the 3D model of the target object is improved.

[0147] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0148] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The apparatus can be mounted on... Figure 1 On the terminal device 101 or server 102 shown. Figure 5 The image processing apparatus shown can be used to perform the above. Figure 2 and Figure 3 Some or all of the functionality described in the method embodiments. Please refer to... Figure 5 The detailed descriptions of each unit are as follows:

[0149] The acquisition unit 501 is used to acquire a multi-viewpoint image set of the target object, which contains multiple viewpoint images; the multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints respectively;

[0150] The processing unit 502 is used to predict the three-dimensional coordinates of the target object in each viewpoint image based on the position information of the target object in each viewpoint image of the multi-viewpoint image set.

[0151] And it is used to calibrate the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, so as to obtain the three-dimensional calibration coordinates of the target object based on the calibration point;

[0152] And it is used to fuse the three-dimensional calibration coordinates of the target object based on the calibration points to obtain the three-dimensional model of the target object.

[0153] In one implementation, the processing unit 501 is used to predict the three-dimensional coordinates of the target object at the viewpoint corresponding to each viewpoint image based on the position information of the target object in each viewpoint image, specifically for:

[0154] The position information of the target object in each viewpoint image of the multi-viewpoint image set is processed by convolution to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0155] In one embodiment, the processing unit 501 is configured to perform convolution processing on the position information of the target object in each viewpoint image of the multi-viewpoint image set to obtain the three-dimensional coordinate points of the target object under the viewpoint corresponding to each viewpoint image, specifically configured to:

[0156] Each viewpoint image is encoded to obtain a binary mask for each viewpoint image;

[0157] The binary mask of each viewpoint image is generated by calling the 3D structure generation model to obtain the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0158] The 3D structure generation model includes a 2D convolutional layer, which is used to perform 2D convolution operations on the binary masks of each viewpoint image.

[0159] In one embodiment, the processing unit 501 is further configured to:

[0160] Obtain the spatial location of the reference viewpoint, which refers to any viewpoint other than the viewpoint corresponding to each viewpoint image in the multi-viewpoint image set;

[0161] Based on the spatial position of the reference viewpoint, the 3D model is projected to obtain the projected image of the 3D model of the target object under the reference viewpoint.

[0162] The projected image of the 3D model at the reference viewpoint is processed by pseudo-rendering to obtain the predicted image of the 3D model at the reference viewpoint.

[0163] Based on the loss value between the predicted image and the labeled image at the reference viewpoint, the parameters in the 3D structure generation model are optimized to obtain the optimized 3D structure generation model.

[0164] In one embodiment, the processing unit 501 is configured to perform pseudo-rendering processing on the projection image of the 3D model from a reference viewpoint to obtain a predicted image of the 3D model from the reference viewpoint, specifically for:

[0165] The projection image of the 3D model at the reference viewpoint is upsampled by a factor of U to obtain the upsampled image, where U is a positive integer.

[0166] The updated upsampled image is downsampled, and the pixel with the smallest depth value among all pixel positions in the downsampled image is retained to obtain the predicted image of the 3D model at the reference viewpoint.

[0167] In one embodiment, the processing unit 501 is configured to optimize the 3D structure generation model based on the loss value between the predicted image and the labeled image at the reference viewpoint, specifically for:

[0168] The depth loss component is calculated based on the difference between the depth image corresponding to the predicted image under the reference viewpoint and the depth image corresponding to the labeled image under the reference viewpoint.

[0169] The mask loss component is calculated based on the difference between the mask image corresponding to the predicted image under the reference viewpoint and the mask image corresponding to the labeled image under the reference viewpoint.

[0170] By using the depth loss component and the mask loss component, the loss value between the predicted image and the labeled image at the reference viewpoint is obtained.

[0171] The parameters in the 3D structure generation model are optimized based on the loss value so that the 3D structure generation model meets the optimization conditions.

[0172] In one embodiment, the processing unit 501 is configured to calibrate the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, thereby obtaining the three-dimensional calibration coordinates of the target object based on the calibration point. Specifically, it is configured to:

[0173] Based on the spatial position of the viewpoint and the spatial position of the calibration point corresponding to each viewpoint image, determine the calibration matrix corresponding to each viewpoint;

[0174] By using the calibration matrix corresponding to each viewpoint, the three-dimensional coordinate points under that viewpoint are calibrated to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0175] In one embodiment, the processing unit 501 is further configured to:

[0176] Construct a 3D point cloud map and obtain the placement information of the 3D model of the target object in the 3D point cloud map;

[0177] In response to the confirmation of the placement location information, the 3D model of the target object is added to the location indicated by the placement location information in the 3D point cloud map.

[0178] In one embodiment, the processing unit 501 is configured to construct a three-dimensional point cloud map, specifically for:

[0179] Obtain the position information of the reference object;

[0180] Based on the position information of the reference object, a reference coordinate system is established. The reference coordinate system is associated with projection information, which is used to indicate the projection relationship between the reference coordinate system and the real world.

[0181] A 3D point cloud map is constructed in a reference coordinate system using projection information.

[0182] In one embodiment, the processing unit 501 is further configured to:

[0183] In response to viewing operations on a 3D point cloud map, obtain the observation position of the 3D point cloud map;

[0184] Based on the observation location, a planar projection is performed on the 3D point cloud map to obtain the observation effect image of the 3D point cloud map at the observation location.

[0185] According to one embodiment of this application, Figure 2 and Figure 3 The image processing method shown may involve some steps that can be derived from... Figure 5 The image processing apparatus shown is executed by each unit. For example, Figure 2 Step S201 shown can be performed by Figure 5 The acquisition unit 501 shown is executed, and steps S202-S204 can be performed by... Figure 5 The processing unit 502 shown is executed. Figure 3 Step S301 shown can be performed by Figure 5 The acquisition unit 501 shown is executed, and steps S302-S310 can be performed by... Figure 5 The processing unit 502 shown is executed. Figure 5 The various units in the image processing apparatus shown can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the image processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0186] According to another embodiment of this application, the following can be executed by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). Figure 2 and Figure 3 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 5 The image processing apparatus shown herein, and the image processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0187] Based on the same inventive concept, the principle and beneficial effects of the image processing apparatus provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the image processing method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0188] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application, such as... Figure 6 As shown, the image processing device includes at least a processor 601, a communication interface 602, and a memory 603. The processor 601, communication interface 602, and memory 603 can be connected via a bus or other means. The processor 601 (or Central Processing Unit, CPU) is the computing and control core of the terminal. It can parse various instructions within the terminal and process various types of data. For example, the CPU can parse power-on / off commands sent by the user to the terminal and control the terminal to perform power-on / off operations; it can also transmit various interactive data between internal structures within the terminal, and so on. The communication interface 602 can optionally include standard wired interfaces and wireless interfaces (such as Wi-Fi, mobile communication interfaces, etc.), and can be used to send and receive data under the control of the processor 601; the communication interface 602 can also be used for data transmission and interaction within the terminal. The memory 603 is the memory device in the terminal, used to store programs and data. It can be understood that the memory 603 here can include the terminal's built-in memory, or it can include extended memory supported by the terminal. The memory 603 provides storage space for storing the terminal's operating system, which may include, but is not limited to, Android, iOS, Windows Phone, etc. This application does not limit this.

[0189] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a terminal for storing programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal and extended storage media supported by the terminal. The computer-readable storage medium provides storage space that stores the terminal's processing system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 601, which can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0190] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor 601 loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above-described image processing method embodiment; specifically, the one or more instructions in the computer-readable storage medium are loaded by the processor 601 and executed as follows:

[0191] The multi-viewpoint image set of the target object is obtained through the communication interface 602. The multi-viewpoint image set contains multiple viewpoint images. The multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints.

[0192] Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, predict the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image;

[0193] Based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, the three-dimensional coordinate points of the target object under each viewpoint are calibrated to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0194] The three-dimensional calibration coordinates of the target object based on the calibration points are fused to obtain the three-dimensional model of the target object.

[0195] As an optional embodiment, the processor 601 predicts the three-dimensional coordinates of the target object at the viewpoint corresponding to each viewpoint image based on the target object's position information in each viewpoint image. A specific embodiment of this is as follows:

[0196] The position information of the target object in each viewpoint image of the multi-viewpoint image set is processed by convolution to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0197] As an optional embodiment, the processor 601 performs convolution processing on the position information of the target object in each viewpoint image of the multi-viewpoint image set to obtain the three-dimensional coordinate points of the target object under the viewpoint corresponding to each viewpoint image. A specific embodiment of this is as follows:

[0198] Each viewpoint image is encoded to obtain a binary mask for each viewpoint image;

[0199] The binary mask of each viewpoint image is generated by calling the 3D structure generation model to obtain the 3D coordinates of the target object under the viewpoint corresponding to each viewpoint image.

[0200] The 3D structure generation model includes a 2D convolutional layer, which is used to perform 2D convolution operations on the binary masks of each viewpoint image.

[0201] As an optional embodiment, the processor 601, by running executable program code in the memory 603, also performs the following operations:

[0202] Obtain the spatial location of the reference viewpoint, which refers to any viewpoint other than the viewpoint corresponding to each viewpoint image in the multi-viewpoint image set;

[0203] Based on the spatial position of the reference viewpoint, the 3D model is projected to obtain the projected image of the 3D model of the target object under the reference viewpoint.

[0204] The projected image of the 3D model at the reference viewpoint is processed by pseudo-rendering to obtain the predicted image of the 3D model at the reference viewpoint.

[0205] Based on the loss value between the predicted image and the labeled image at the reference viewpoint, the parameters in the 3D structure generation model are optimized to obtain the optimized 3D structure generation model.

[0206] As an optional embodiment, the processor 601 performs pseudo-rendering processing on the projected image of the 3D model at the reference viewpoint to obtain a predicted image of the 3D model at the reference viewpoint. A specific embodiment of this is as follows:

[0207] The projection image of the 3D model at the reference viewpoint is upsampled by a factor of U to obtain the upsampled image, where U is a positive integer.

[0208] The updated upsampled image is downsampled, and the pixel with the smallest depth value among all pixel positions in the downsampled image is retained to obtain the predicted image of the 3D model at the reference viewpoint.

[0209] As an optional embodiment, the processor 601 optimizes the 3D structure generation model based on the loss value between the predicted image and the labeled image at the reference viewpoint. A specific embodiment of this optimization is as follows:

[0210] The depth loss component is calculated based on the difference between the depth image corresponding to the predicted image under the reference viewpoint and the depth image corresponding to the labeled image under the reference viewpoint.

[0211] The mask loss component is calculated based on the difference between the mask image corresponding to the predicted image under the reference viewpoint and the mask image corresponding to the labeled image under the reference viewpoint.

[0212] By using the depth loss component and the mask loss component, the loss value between the predicted image and the labeled image at the reference viewpoint is obtained.

[0213] The parameters in the 3D structure generation model are optimized based on the loss value so that the 3D structure generation model meets the optimization conditions.

[0214] As an optional embodiment, the processor 601 calibrates the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, and obtains the three-dimensional calibration coordinates of the target object based on the calibration point. A specific embodiment of this is as follows:

[0215] Based on the spatial position of the viewpoint and the spatial position of the calibration point corresponding to each viewpoint image, determine the calibration matrix corresponding to each viewpoint;

[0216] By using the calibration matrix corresponding to each viewpoint, the three-dimensional coordinate points under that viewpoint are calibrated to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

[0217] As an optional embodiment, the processor 601, by running executable program code in the memory 603, also performs the following operations:

[0218] Construct a 3D point cloud map and obtain the placement information of the 3D model of the target object in the 3D point cloud map;

[0219] In response to the confirmation of the placement location information, the 3D model of the target object is added to the location indicated by the placement location information in the 3D point cloud map.

[0220] As an optional embodiment, the processor 601 constructs a 3D point cloud map in the following specific embodiment:

[0221] Obtain the position information of the reference object;

[0222] Based on the position information of the reference object, a reference coordinate system is established. The reference coordinate system is associated with projection information, which is used to indicate the projection relationship between the reference coordinate system and the real world.

[0223] A 3D point cloud map is constructed in a reference coordinate system using projection information.

[0224] As an optional embodiment, the processor 601, by running executable program code in the memory 603, also performs the following operations:

[0225] In response to viewing operations on a 3D point cloud map, obtain the observation position of the 3D point cloud map;

[0226] Based on the observation location, a planar projection is performed on the 3D point cloud map to obtain the observation effect image of the 3D point cloud map at the observation location.

[0227] Based on the same inventive concept, the principle and beneficial effects of the image processing device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the image processing method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, they will not be repeated here.

[0228] This application also provides a computer-readable storage medium storing one or more instructions adapted for loading by a processor and executing the image processing method described in the above-described method embodiments.

[0229] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the image processing method described in the above method embodiments.

[0230] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image processing method described above.

[0231] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0232] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0233] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0234] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: A multi-viewpoint image set of a target object is obtained, wherein the multi-viewpoint image set contains multiple viewpoint images; the multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints respectively; Based on the position information of the target object in each viewpoint image of the multi-viewpoint image set, predict the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image; Based on the positional relationship between the viewpoints and calibration points corresponding to each viewpoint image, the three-dimensional coordinates of the target object under each viewpoint are calibrated to obtain the three-dimensional calibration coordinates of the target object based on the calibration points. Here, the position of the viewpoint corresponding to each viewpoint image is the spatial position of the target object when the image acquisition device captures the image; the calibration point is the spatial position of one viewpoint among the viewpoints corresponding to each viewpoint image; the prediction of the three-dimensional coordinates of the target object under the viewpoints corresponding to each viewpoint image, and the calibration of the three-dimensional coordinates of the target object under each viewpoint, are both determined solely based on each viewpoint image in the multi-viewpoint image set; the three-dimensional calibration coordinates of the target object based on the calibration points are the absolute positioning coordinates of the three-dimensional coordinates of the target object under each viewpoint relative to the calibration points. The three-dimensional calibration coordinates of the target object based on the calibration point are fused to obtain the three-dimensional model of the target object.

2. The method as described in claim 1, characterized in that, The step of predicting the three-dimensional coordinates of the target object at the corresponding viewpoint in each viewpoint image of the multi-viewpoint image set, based on the position information of the target object in each viewpoint image, includes: The position information of the target object in each viewpoint image of the multi-viewpoint image set is convolved to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image.

3. The method as described in claim 2, characterized in that, The step of convolving the position information of the target object in each viewpoint image of the multi-viewpoint image set to obtain the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image includes: Each viewpoint image is encoded to obtain a binary mask for each viewpoint image; The binary mask of each viewpoint image is generated by calling the 3D structure generation model to obtain the 3D coordinate points of the target object under the viewpoint corresponding to each viewpoint image; The three-dimensional structure generation model includes a two-dimensional convolutional layer, which is used to perform two-dimensional convolution operations on the binary masks of each viewpoint image.

4. The method as described in claim 3, characterized in that, The method further includes: Obtain the spatial location of a reference viewpoint, where the reference viewpoint refers to any viewpoint other than the viewpoint corresponding to each viewpoint image in the multi-viewpoint image set; Based on the spatial position of the reference viewpoint, the three-dimensional model is projected to obtain a projected image of the three-dimensional model of the target object under the reference viewpoint; The projection image of the 3D model under the reference viewpoint is subjected to pseudo-rendering processing to obtain the predicted image of the 3D model under the reference viewpoint. Based on the loss value between the predicted image under the reference viewpoint and the labeled image under the reference viewpoint, the parameters in the 3D structure generation model are optimized to obtain the optimized 3D structure generation model.

5. The method as described in claim 4, characterized in that, The step of performing pseudo-rendering processing on the projected image of the 3D model under the reference viewpoint to obtain the predicted image of the 3D model under the reference viewpoint includes: The projection image of the three-dimensional model under the reference viewpoint is upsampled by a factor of U to obtain an upsampled image, where U is a positive integer; The updated upsampled image is downsampled, and the pixel with the smallest depth value among all pixel positions in the downsampled image is retained to obtain the predicted image of the 3D model at the reference viewpoint.

6. The method as described in claim 4, characterized in that, The step of optimizing the 3D structure generation model based on the loss value between the predicted image and the labeled image at the reference viewpoint includes: The depth loss component is calculated based on the difference between the depth image corresponding to the predicted image under the reference viewpoint and the depth image corresponding to the labeled image under the reference viewpoint. The mask loss component is calculated based on the difference between the mask image corresponding to the predicted image under the reference viewpoint and the mask image corresponding to the labeled image under the reference viewpoint. The loss value between the predicted image and the labeled image at the reference viewpoint is obtained by using the depth loss component and the mask loss component. The parameters in the 3D structure generation model are optimized based on the loss value so that the 3D structure generation model meets the optimization conditions.

7. The method as described in claim 1, characterized in that, The step of calibrating the three-dimensional coordinates of the target object at each viewpoint based on the positional relationship between the viewpoint and calibration point corresponding to each viewpoint image, to obtain the three-dimensional calibration coordinates of the target object based on the calibration point, includes: Based on the spatial position of the viewpoint and the spatial position of the calibration point corresponding to each viewpoint image, determine the calibration matrix corresponding to each viewpoint; By using the calibration matrix corresponding to each viewpoint, the three-dimensional coordinate points under that viewpoint are calibrated to obtain the three-dimensional calibration coordinate points of the target object based on the calibration points.

8. The method as described in claim 1, characterized in that, The method further includes: Construct a 3D point cloud map and obtain the placement information of the 3D model of the target object in the 3D point cloud map; In response to the confirmation of the placement location information, the 3D model of the target object is added to the location indicated by the placement location information in the 3D point cloud map.

9. The method as described in claim 8, characterized in that, The construction of the 3D point cloud map includes: Obtain the position information of the reference object; Based on the position information of the reference object, a reference coordinate system is established. The reference coordinate system is associated with projection information, which is used to indicate the projection relationship between the reference coordinate system and the real world. A three-dimensional point cloud map is constructed in the reference coordinate system using the projection information.

10. The method as described in claim 8, characterized in that, The method further includes: In response to the viewing operation of the 3D point cloud map, the observation position of the 3D point cloud map is obtained; Based on the observation location, the three-dimensional point cloud map is projected onto a plane to obtain an observation effect image of the three-dimensional point cloud map at the observation location.

11. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire a multi-viewpoint image set of the target object, wherein the multi-viewpoint image set contains multiple viewpoint images; The multiple viewpoint images are obtained by taking pictures of the target object from multiple viewpoints respectively; The processing unit is used to predict the three-dimensional coordinates of the target object under the viewpoint corresponding to each viewpoint image based on the position information of the target object in each viewpoint image of the multi-viewpoint image set. And it is used to calibrate the three-dimensional coordinates of the target object under each viewpoint according to the positional relationship between the viewpoint and the calibration point corresponding to each viewpoint image, so as to obtain the three-dimensional calibration coordinates of the target object based on the calibration point; wherein, the position of the viewpoint corresponding to each viewpoint image is the spatial position of the target object when the image acquisition device captures the target object, the calibration point is the spatial position of one viewpoint in the viewpoints corresponding to each viewpoint image, the prediction of the three-dimensional coordinates of the target object under the viewpoints corresponding to each viewpoint image, and the calibration of the three-dimensional coordinates of the target object under each viewpoint are determined only based on each viewpoint image in the multi-viewpoint image set, and the three-dimensional calibration coordinates of the target object based on the calibration point are the absolute positioning coordinates of the three-dimensional coordinates of the target object under each viewpoint relative to the calibration point; And to fuse the three-dimensional calibration coordinates of the target object based on the calibration point to obtain a three-dimensional model of the target object.

12. An image processing device, characterized in that, include: Storage devices and processors; The storage device stores a computer program; A processor that executes a computer program to implement the image processing method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image processing method as described in any one of claims 1-10.

Images