Data processing method, device, storage medium and computer equipment

Abstract

The present invention discloses a data processing method, apparatus, storage medium, and computer equipment. The method comprises: acquiring a color image and a depth image of a physical target object; fusing multiple depth images to obtain a three-dimensional reconstruction model of the physical target object; determining a first pose of a virtual target object in a predetermined coordinate system and a second pose of a virtual object in the predetermined coordinate system based on the three-dimensional reconstruction model; projecting 3D key points of the virtual target object onto an image plane to obtain 2D key points based on the first pose; and rendering an interactive image of the virtual object and the physical target object in the image plane based on the second pose and the color image. The present invention solves the technical problems of high cost and low quality of training data acquisition.

Description

Technical Field

[0001] The present invention relates to the field of images, and in particular to a data processing method, device, storage medium and computer equipment. Background Art

[0002] Augmented Reality (AR) shoe fitting is a technology based on computer vision and graphics, which can provide new retail solutions for the footwear industry. In the technology of AR shoe fitting, supervised learning methods are usually used to detect the key points of the foot in 2D images, but this supervised learning method requires rich training data to enhance the accuracy and robustness of the detection algorithm. The training data includes the shoe fitting image and the pixel positions of the 2D foot key points in the image. In the related art, the training data is obtained by using a fully manual method of frame-by-frame annotation, a semi-automatic method of annotating key frames, or a fully automatic virtual rendering method. However, there are many problems with the above methods of obtaining training data. For example, the manual annotation method has the problems of unstable cost and uncontrollable data quality, while the fully automatic virtual rendering method has the problems of lack of authenticity of the generated data and the tendency of samples to be identical, resulting in low training value.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] Embodiments of the present invention provide a data processing method, apparatus, storage medium, and computer equipment to at least solve the technical problems of high cost of obtaining training data and low quality of training data.

[0005] According to one aspect of an embodiment of the present invention, a data processing method is provided, comprising: acquiring a color image and a depth image of a physical target object; determining a three-dimensional reconstruction model of the physical target object based on the depth image; determining a first pose of a virtual target object in a predetermined coordinate system based on the three-dimensional reconstruction model, and determining a second pose of a virtual object in the predetermined coordinate system; projecting 3D key points of the virtual target object onto an image plane to obtain 2D key points based on the first pose, and drawing an interaction image of the virtual object and the physical target object in the image plane based on the second pose and the color image.

[0006] Optionally, in the case where there are multiple depth images, the three-dimensional reconstruction model of the physical target object is determined based on the multiple depth images, including: converting the multiple depth images into multiple three-dimensional point clouds; establishing a point cloud pose graph based on the multiple three-dimensional point clouds, wherein each point cloud in the multiple three-dimensional point clouds is a node in the point cloud pose graph; and aligning the point cloud pose graph to obtain the three-dimensional reconstruction model of the physical target object.

[0007] Optionally, the point cloud pose graph is registered to obtain a three-dimensional reconstructed model of the physical target object, including: using an iterative closest point ICP method to serialize the point cloud pose graph to obtain a serialized registration result graph of the point cloud pose graph; and globally registering the serialized registration result graph to obtain a three-dimensional reconstructed model of the physical target object, wherein the pose of each point cloud in the three-dimensional reconstructed model is a pose in the global coordinate system.

[0008] Optionally, based on the three-dimensional reconstructed model, determining the first pose of the virtual target object in a predetermined coordinate system includes: obtaining a third pose of the three-dimensional reconstructed model in the predetermined coordinate system; determining a transformation matrix of the virtual target object to the three-dimensional reconstructed model; and determining the first pose of the virtual target object in the predetermined coordinate system based on the third pose and the transformation matrix.

[0009] Optionally, determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model includes: aligning the virtual target object with the three-dimensional reconstructed model based on a point cloud coarse registration method of a fast point feature histogram FPFH and a point cloud fine registration method of a point-to-surface ICP to obtain alignment parameters; and determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model based on the alignment parameters.

[0010] Optionally, determining a second posture of the virtual object in the predetermined coordinate system includes: obtaining an alignment relationship between the virtual object and the virtual target object; and determining the second posture of the virtual object in the predetermined coordinate system based on the first posture and the alignment relationship.

[0011] Optionally, according to the first posture, the 3D key points of the virtual target object are projected onto the image plane to obtain 2D key points, and according to the second posture and the color image, the interaction image of the virtual object and the physical target object in the image plane is drawn, including: according to the first posture and a predetermined imaging method, the 3D key points of the virtual target object are projected onto the image plane to obtain 2D key points; according to the second posture and the color image, a predetermined rendering engine is called to draw the result of the interaction between the virtual object and the physical target object in the image plane, so as to obtain the interaction image of the virtual object and the physical target object in the image plane.

[0012] Optionally, based on the second posture and the color image, a predetermined rendering engine is called to draw the result of the interaction between the virtual object and the physical target object in the image plane to obtain an interaction image between the virtual object and the physical target object in the image plane, including: when calling the predetermined rendering engine to draw the interaction result between the virtual object and the physical target object in the image plane based on the second posture and the color image, multiple interaction images between the virtual object and the physical target object in the image plane are obtained by replacing the background and / or adjusting the light source.

[0013] Optionally, when the collected color images and depth images are multiple color images and multiple depth images of the physical target object in different states and / or at different angles, 2D key points of the physical target object in different states and / or at different angles and interaction images with the virtual object are obtained respectively.

[0014] Optionally, training data is generated based on the 2D key points and the interaction image; different training data corresponding to multiple physical target objects are obtained; and machine training is performed using the different training data corresponding to multiple physical target objects to obtain a key point detection model.

[0015] According to another aspect of an embodiment of the present invention, a data processing method is also provided, including: receiving an input image, wherein the input image includes a physical target object; using a key point detection model to detect 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second pose of the virtual target object in the predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to a three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing multiple depth images of the collected physical target object.

[0016] According to another aspect of an embodiment of the present invention, a data processing method is also provided, including: obtaining multiple sets of training data, wherein the data in the multiple sets of training data include: an interaction image between a physical target object and a virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto an image plane according to a first posture of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to a second posture of the virtual object in the predetermined coordinate system and a collected color image of the physical target object, the first posture and the second posture are determined according to a three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing multiple depth images of the collected physical target object; using the multiple sets of training data for machine training to obtain a key point detection model.

[0017] According to another aspect of an embodiment of the present invention, a data processing device is further provided, including: an acquisition module for acquiring a color image and a depth image of a physical target object; a first processing module for fusing the multiple depth images to obtain a three-dimensional reconstruction model of the physical target object; a determination module for determining a first pose of the virtual target object in a predetermined coordinate system and a second pose of the virtual object in the predetermined coordinate system based on the three-dimensional reconstruction model; a second processing module for projecting the 3D key points of the virtual target object onto an image plane to obtain 2D key points based on the first pose, and drawing an interaction image of the virtual object and the physical target object in the image plane based on the second pose and the color image.

[0018] According to another aspect of an embodiment of the present invention, a data processing device is further provided, including: a receiving module for receiving an input image, wherein the input image includes a physical target object; a detection module for using a key point detection model to detect 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual target object on the image plane according to the second pose of the virtual target object in the predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to a three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing multiple depth images of the collected physical target object.

[0019] According to another aspect of an embodiment of the present invention, a data processing device is further provided, including: an acquisition module for acquiring multiple sets of training data, wherein the data in the multiple sets of training data include: an interaction image between a physical target object and a virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto an image plane according to a first posture of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to a second posture of the virtual object in the predetermined coordinate system and a collected color image of the physical target object, the first posture and the second posture are determined according to a three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing multiple depth images of the collected physical target object; a training module for performing machine training using the multiple sets of training data to obtain a key point detection model.

[0020] According to another aspect of the embodiments of the present invention, a storage medium is provided, which includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute any one of the above-mentioned data processing methods.

[0021] According to another aspect of an embodiment of the present invention, a computer device is provided, comprising: a memory and a processor, wherein the memory stores a computer program; and the processor is configured to execute the computer program stored in the memory, wherein when the computer program is executed, the processor executes any one of the above-described data processing methods.

[0022] According to another aspect of an embodiment of the present invention, a data processing method is also provided, including: acquiring a color image and a depth image of a foot; determining a three-dimensional reconstructed foot model of the foot based on the depth image; determining a first posture of the virtual foot in a predetermined coordinate system based on the three-dimensional reconstructed foot model, and determining a second posture of the virtual shoe in the predetermined coordinate system; projecting the 3D foot key points of the virtual foot onto an image plane to obtain 2D foot key points based on the first posture, and drawing a fitting image of the virtual shoe on the image plane based on the second posture and the color image.

[0023] In an embodiment of the present invention, a method of collecting color images and depth images of a physical target object is adopted, and a three-dimensional reconstruction model of the physical target object is obtained by fusing multiple depth images. According to the three-dimensional reconstruction model, the first pose of the virtual target object in a predetermined coordinate system is determined, and the second pose of the virtual object in the predetermined coordinate system is determined. According to the first pose, the 3D key points of the virtual target object are projected onto the image plane to obtain 2D key points, and according to the second pose and the color image, an interactive image of the virtual object and the physical target object in the image plane is drawn. The purpose of generating an interactive image marked with 2D key points according to the image of the physical target object is achieved, thereby achieving the technical effect of reducing the cost of obtaining training data and improving the quality of training data, and further solving the technical problems of high cost of obtaining training data and low quality of training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0025] Figure 1 is a block diagram of a computer terminal hardware structure for implementing a data processing method according to an embodiment of the present invention;

[0026] Figure 2 is a flow chart of a data processing method 1 according to embodiment 1 of the present invention;

[0027] Figure 3 is a flow chart of a second data processing method according to embodiment 1 of the present invention;

[0028] Figure 4 is a flow chart of a data processing method 3 according to embodiment 1 of the present invention;

[0029] Figure 5 is a flow chart of a fourth data processing method according to embodiment 1 of the present invention;

[0030] Figure 6 is a schematic diagram of a training data acquisition process according to an optional embodiment of the present invention;

[0031] Figure 7 is a schematic diagram of obtaining a three-dimensional reconstructed model according to an optional embodiment of the present invention;

[0032] Figure 8 is a schematic diagram of aligning a virtual shoe, a virtual foot, and a reconstructed foot according to an optional embodiment of the present invention;

[0033] Figure 9 is a structural block diagram of a data processing device according to embodiment 2 of the present invention;

[0034] Figure 10 is a structural block diagram of a data processing device 2 according to embodiment 3 of the present invention;

[0035] Figure 11 is a structural block diagram of a data processing device 3 according to embodiment 4 of the present invention;

[0036] Figure 12 is a block diagram of the structure of a data processing device 4 according to embodiment 5 of the present invention;

[0037] Figure 13 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0039] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0041] Deep Learning: Deep learning, also known as model training, refers to a collection of machine learning algorithms applied to multi-layer neural networks to solve various problems involving images, text, and other data. Deep learning can generally be categorized as a neural network, but its specific implementation varies greatly. The core of deep learning is feature learning, which aims to obtain hierarchical feature information through layered networks, thereby solving important problems that previously required manual feature design.

[0042] 6DoF: 6 degrees of freedom, which refers to the translation of an object along the x, y, and z axes and the rotation around the axis in space.

[0043] PnP: Perspective-n-Point, n-point perspective (given 2D-3D point pairs, solve 6DoF pose).

[0044] ICP: Iterative closest point, iterative closest point registration algorithm, for paired point cloud registration.

[0045] Pose Graph Optimization: Pose graph optimization for global point cloud registration.

[0046] 3D key points: multiple key points on the 3D model of the virtual target object.

[0047] 2D keypoint: The 2D projection point of the 3D keypoint on the imaging plane.

[0048] Virtual target object: a 3D model of the target object created by an artist.

[0049] Virtual Object: A 3D model of an object created by an artist that is parametrically aligned with a virtual target object.

[0050] Example 1

[0051] According to an embodiment of the present invention, an embodiment of a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] The method embodiment provided in Example 1 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more (shown as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0053] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0054] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the vulnerability detection method of the application program described above. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0055] The transmission device is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0056] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0057] Under the above operating environment, this application provides Figure 2 The data processing method shown. Figure 2 1 is a flow chart of a data processing method according to embodiment 1 of the present invention. Figure 2 As shown, the method includes the following steps:

[0058] Step S202, collecting a color image and a depth image of a physical target object;

[0059] Step S204: fusing the multiple depth images to obtain a three-dimensional reconstructed model of the physical target object;

[0060] Step S206, determining a first pose of the virtual target object in a predetermined coordinate system and a second pose of the virtual object in the predetermined coordinate system based on the three-dimensional reconstructed model;

[0061] Step S208 : projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points according to the first pose, and drawing an interactive image of the virtual object and the physical target object in the image plane according to the second pose and the color image.

[0062] Through the above steps, the purpose of establishing a three-dimensional reconstruction model based on the image of the physical target object is achieved, and an interactive image of the physical target object and the virtual object with 2D key points marked is generated based on the established three-dimensional reconstruction model, thereby achieving the technical effect of reducing the cost of obtaining training data and improving the quality of training data, and thus solving the technical problems of high cost and low quality of training data.

[0063] As an optional embodiment, in the case of collecting images of physical target objects, the acquisition of the images of physical target objects can be obtained by camera shooting, vector graphics, etc. The above method can be used to accurately generate interactive images of virtual objects including 2D key points on the image plane and the physical target object, and the above images can be used as training data to train the key point detection model. The interactive images of the virtual objects on the physical target objects obtained according to this embodiment are labeled with 2D key points, which avoids the problems of inefficiency and difficulty in controlling the error rate when manually labeling 2D key points. At the same time, since the real data of the physical target object is used in the process of generating the interactive images of the virtual object and the physical target object, the generated interactive images of the virtual object and the physical target object are more realistic, and can achieve better training effects as training data.

[0064] As an optional embodiment, when collecting color images and depth images of a physical target object, it is sufficient as long as the information of the physical target object can be obtained from multiple angles. Therefore, the number of collected color images can be one or more, and the number of depth images can also be one or more. For example, when using the color image of a physical target object, if a panoramic camera is used to shoot the physical target object, information from multiple perspectives of the physical target object can be obtained; and if a non-panoramic camera is used to shoot the physical target object, the physical target object can be shot multiple times from multiple angles to obtain information from multiple perspectives of the physical target object. When using a depth image, the same processing method as that for collecting color images can also be used. For example, collecting color images and collecting depth images can be performed simultaneously.

[0065] As an optional embodiment, when the collected color images and depth images are multiple color images and multiple depth images of the physical target object in different states and / or at different angles, 2D key points and interaction images of the physical target object in different states and / or at different angles are obtained. By using image data of multiple physical target objects and obtaining accurate 2D key points through the correspondence between the three-dimensional reconstruction model, and by drawing the interaction images of the virtual object and the physical target object in different states (for example, when the physical target object is a foot, different states of the foot include barefoot or wearing socks), different types, different postures, and different perspectives, the interaction images of the physical target object and the virtual object can provide accurate and diverse training data for the subsequent training process of the key point monitoring model, thereby compensating for the problem that the interaction images of the virtual object and the physical target object in the image plane used as training data often lack certain special perspectives. For example, images of wearing shoes from the toe perspective, the heel perspective, and the side-on perspective are very rare in existing image training sets. This embodiment can effectively compensate for the lack of such training data and provide richer training materials.

[0066] Point cloud data can obtain the accurate topological and geometric structure of an object at a relatively low storage cost. At the same time, when photographing a physical target object, a single scan may not easily obtain the complete geometric information of the physical target object, and the physical target object needs to be photographed multiple times from different angles. Therefore, a three-dimensional reconstruction model corresponding to the physical target object can be established by aligning multiple groups of point clouds corresponding to multiple images of the physical target object. As an optional embodiment, in the case of multiple depth images, the three-dimensional reconstruction model of the physical target object obtained by fusing the multiple depth images can be achieved by the following optional method: converting the multiple depth images into multiple three-dimensional point clouds, establishing a point cloud pose graph based on the multiple three-dimensional point clouds, wherein each point cloud in the multiple three-dimensional point clouds is a node in the point cloud pose graph, and then aligning the point cloud pose graph to obtain a three-dimensional reconstruction model of the physical target object.

[0067] As an optional embodiment, the above-mentioned predetermined coordinate system may be of multiple types. For example, the above-mentioned depth image is obtained by the same shooting, and there are multiple camera coordinate systems, such as the pixel plane coordinate system (u, v), the image plane coordinate system, the image physical coordinate system (x, y), the camera coordinate system (Xc, Yc, Zc) and the world coordinate system. The camera coordinate system is used here. Camera coordinate system: a three-dimensional rectangular coordinate system with the focusing center of the camera as the origin and the optical axis as the Z axis. The origin of the camera coordinate system is the optical center of the camera, the x-axis and the y-axis are parallel to the X and Y axes of the image, and the z-axis is the optical axis of the camera, which is perpendicular to the graphic plane. The intersection of the optical axis and the image plane is the origin of the image coordinate system, and the image coordinate system is a two-dimensional rectangular coordinate system.

[0068] As an optional embodiment, the point cloud pose graph can be registered in the following manner to obtain a three-dimensional reconstruction model of the physical target object: using the iterative closest point (ICP) method, the point cloud pose graph is serialized and registered to obtain a serialized registration result graph of the point cloud pose graph; and the serialized registration result graph is globally registered to obtain a three-dimensional reconstruction model of the physical target object, wherein the pose of each point cloud in the three-dimensional reconstruction model is the pose in the global coordinate system. Since the perspectives and occlusion relationships of the images obtained by photographing the physical target object are different, the point clouds corresponding to the images are not in the same coordinate system. By serializing and globally registering the point cloud pose graph, the pose data of the physical target object obtained from different shooting angles can be transformed into the same predetermined coordinate system, thereby realizing the construction of a three-dimensional reconstruction model of the physical target object.

[0069] As an optional embodiment, when obtaining a three-dimensional reconstruction model of a physical target object using multiple depth images, since each of the multiple depth images has its own corresponding camera coordinate system, in order to establish an accurate three-dimensional reconstruction model, a unified coordinate system can be selected for the multiple depth images for global registration of the multiple depth images. There are many ways to select this unified coordinate system. For example, the camera coordinate system when the first depth image was taken can be selected as the unified coordinate system, or the camera coordinate system corresponding to the depth images taken at other time points can be selected as the unified coordinate system. The selection can be flexibly based on needs.

[0070] As an optional embodiment, determining the first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstruction model can be achieved as follows: obtaining the third pose of the three-dimensional reconstruction model in the predetermined coordinate system; determining the transformation matrix of the virtual target object to the three-dimensional reconstruction model; and determining the first pose of the virtual target object in the predetermined coordinate system based on the third pose and the transformation matrix. Specifically, when the virtual target object and the three-dimensional reconstruction model are already aligned, the transformation matrix of the virtual target object and the three-dimensional reconstruction model can be obtained. The third pose of the three-dimensional reconstruction model in the predetermined coordinate system can be solved in the process of global registration of the three-dimensional reconstruction model. Therefore, the virtual target object can be transformed by the transformation matrix based on the three-dimensional reconstruction model to obtain the pose of the virtual target object in the predetermined coordinate system.

[0071] Fast Point Feature Histograms (FPFH) are a simplified form of the Point Feature Histograms (PFH) calculation method. It calculates the simplified point feature histogram of each point in the k-neighborhood of the query point separately, and then weights all SPFHs into the final fast point feature histogram through a formula. FPFH can be used for registration between three-dimensional point clouds. The Iterative Closest Point (ICP) algorithm can find the nearest points in the matching target point cloud and source point cloud according to certain constraints, and then calculate the optimal matching parameters to minimize the error function. Both the FPFH method and the ICP method can achieve registration between three-dimensional point clouds.

[0072] As an optional embodiment, a coarse point cloud registration method using the Fast Point Feature Histogram (FPFH) and a fine point cloud registration method using the Point-to-Plane (ICP) method can be used to align the virtual target object with the 3D reconstructed model, obtaining alignment parameters. Based on the alignment parameters, a transformation matrix is ​​then determined for transforming the virtual target object to the 3D reconstructed model. The transformation matrix can be used to transform the virtual target object, obtaining its position and pose in a predetermined coordinate system.

[0073] As an optional embodiment, after determining the pose of the 3D reconstructed model and aligning the virtual target object with the 3D reconstructed model, alignment of the virtual object, the virtual target object, and the 3D reconstructed model is achieved based on the alignment relationship between the virtual object and the virtual target object. The alignment relationship between the virtual target object and the virtual object can be predefined. After the virtual target object is aligned with the 3D reconstructed model, the data of the virtual object is adjusted based on the predefined alignment relationship to achieve alignment between the virtual object and the 3D reconstructed model, thereby obtaining the pose of the virtual object in a predetermined coordinate system.

[0074] As an optional embodiment, the second pose of the virtual object in the predetermined coordinate system can be determined by: obtaining the alignment relationship between the virtual object and the virtual target object; and determining the second pose of the virtual object in the predetermined coordinate system based on the first pose and the alignment relationship.

[0075] The training data of the key point monitoring model includes 2D key points and interaction images of virtual objects. As an optional embodiment, the above training data can be obtained in the following way: according to the first posture, the 3D key points of the virtual target object are projected onto the image plane to obtain 2D key points; and according to the second posture and color image, the interaction image of the virtual object and the physical target object in the image plane can be drawn. Specifically, the key points on the 2D image can be obtained in the following way: according to the first posture and the camera imaging method, the 3D key points of the virtual target object are projected onto the image plane to obtain 2D key points. The interaction image of the virtual object and the physical target object in the image plane can also be obtained in the following way: according to the second posture and the color image, a predetermined rendering engine is called to draw the interaction result of the virtual object and the physical target object in the image plane to obtain the interaction image of the virtual object and the physical target object in the image plane.

[0076] As an optional implementation, the 3D key points of the virtual target object can be projected onto the image plane based on the principle of perspective to obtain 2D key points. The obtained image is labeled with key points and does not require manual labeling. The labeling effect is accurate and greatly saves manpower.

[0077] For example, a predefined rendering engine, such as Blender, can be invoked to render the virtual object's interaction with the physical target object in the image plane based on the virtual object's second pose in a predetermined coordinate system. Because the virtual object is already aligned with the 3D reconstructed model generated based on the physical target object, the resulting interaction image generated based on the virtual object and the physical target object is highly realistic, enhancing the authenticity of the generated interaction image between the virtual object and the physical target object in the image plane. Furthermore, when using the rendering engine to render the interaction image between the virtual object and the physical target object in the image plane, multiple images of the virtual object interacting with the physical target object in the image plane can be obtained by replacing the background and / or adjusting the light source. By replacing the background and / or adjusting the light source, the available image material of the virtual object interacting with the physical target object can be greatly enriched. This not only avoids the extensive workload of measuring the physical target object and generating images of the virtual object interacting with the physical target object, but also provides a variety of image materials including different types of interaction environments with the physical target object, enriching the training database and achieving better training results for the keypoint detection model.

[0078] As an optional embodiment, training data is generated based on 2D key points and interactive images; different training data corresponding to a variety of physical target objects are obtained; and machine training is performed using different training data corresponding to a variety of physical target objects to obtain a key point detection model. Through this embodiment, the purpose of using rich training data including 2D key points and interactive images to train the key point detection model can be achieved. The 2D key points in the training data used do not rely on subjective judgment of manual annotation. Taking the SOTA detection results of the key point detection model as the standard, the 2D key point annotation accuracy of the method provided by this embodiment can reach more than 90%. In addition, the key point detection model can be helped to achieve better training results by enriching the training database.

[0079] As an optional embodiment, the physical target objects and virtual objects may include any combination of the following: feet and virtual shoes; head and virtual hat; wrist and virtual bracelet; human body and virtual clothing; face and virtual facial accessories. Different target objects and virtual objects can interact in different ways in different scenarios, for example, generating an image of a foot trying on a virtual shoe, or a head trying on a virtual hat.

[0080] Figure 3 is a flow chart of a second data processing method according to embodiment 1 of the present invention. Figure 3 As shown, the method includes the following steps:

[0081] Step S302: receiving an input image, wherein the input image includes a physical target object;

[0082] In step S304, a key point detection model is used to detect 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second pose of the virtual object in the predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to the three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object.

[0083] Through the above steps, since the key point detection model used is obtained by machine training using a large amount of training data, and the training data is accurately labeled with 2D key points, and the interactive images in the training data are very realistic and diverse in type, the technical effect of accurately detecting the 2D key points of the physical target objects in the received input image including the physical target objects using the key point detection model is achieved, and the technical problem of inaccurate detection of the 2D key points of the physical target objects in the input image is solved.

[0084] Figure 4 : is a flow chart of a data processing method 3 according to embodiment 1 of the present invention. Figure 4 As shown, the method includes the following steps:

[0085] Step S402: Acquire multiple sets of training data, wherein the data in the multiple sets of training data include: an interaction image between a physical target object and a virtual object and 2D key points in the interaction image, wherein the 2D key points are obtained by projecting 3D key points of the virtual target object onto an image plane based on a first pose of the virtual target object in a predetermined coordinate system, and the interaction image is obtained by rendering the virtual object on the image plane based on a second pose of the virtual object in the predetermined coordinate system and a collected color image of the physical target object, wherein the first pose and the second pose are determined based on a 3D reconstruction model, and the 3D reconstruction model is obtained by fusing the collected depth image of the physical target object;

[0086] Step S404: Perform machine training using multiple sets of training data to obtain a key point detection model.

[0087] Through the above steps, we achieved the goal of using multiple sets of training data for machine training to develop a keypoint detection model. Because the training data contained accurate 2D keypoint annotations and the interactive images in it were realistic and diverse, using this training data achieved excellent training results. This solved the technical problem of unsatisfactory training results for 2D keypoint detection models, which often arises from limited training material and inaccurate manual annotation.

[0088] Figure 5 : is a flow chart of a data processing method 4 according to embodiment 1 of the present invention. Figure 5 As shown, the method includes the following steps:

[0089] Step S502, collecting a color image and a depth image of the foot;

[0090] Step S504, determining a three-dimensional reconstructed foot model of the foot according to the depth image;

[0091] Step S506, determining a first pose of the virtual foot in a predetermined coordinate system and a second pose of the virtual shoe in a predetermined coordinate system based on the three-dimensional reconstructed foot model;

[0092] Step S508 : According to the first posture, the 3D foot key points of the virtual foot are projected onto the image plane to obtain 2D foot key points, and according to the second posture and the color image, a fitting image of the virtual shoe is drawn on the image plane.

[0093] Through the above steps, the purpose of establishing a three-dimensional reconstruction model based on the image of the foot is achieved, and an image of the virtual shoe being tried on on the image plane with 2D key points marked is generated based on the established three-dimensional reconstruction model, thereby achieving the technical effect of reducing the cost of obtaining training data and improving the quality of training data, and thus solving the technical problems of high cost and low quality of training data.

[0094] As an optional implementation, the following description will be made using the foot as an example.

[0095] Figure 6 FIG. 1 is a flow chart of training data acquisition according to an optional embodiment of the present invention. Figure 6 As shown in Figure 1, training data including virtual shoe fitting images and 2D foot key points can be obtained by the following steps:

[0096] S1, based on the RGB color data and depth data of the real foot collected by the RGBD camera, performs foot modeling to obtain a 3D foot model of the real foot;

[0097] S2, aligning the virtual foot model according to the 3D foot model;

[0098] S3, aligning the virtual shoe model with the virtual foot model based on the virtual foot model aligned with the 3D foot model;

[0099] S4, determining 2D foot key points obtained by projecting the 3D foot key points on the virtual foot model onto the image according to a rule of projecting the virtual foot model onto the camera image;

[0100] S5, using a rendering engine to render the virtual shoe model and the RGB image of the real foot to obtain a virtual shoe fitting image, wherein the virtual shoe fitting image also includes annotations of 2D foot key points.

[0101] Figure 7 FIG. 1 is a flow chart of the real foot 3D registration according to an optional embodiment of the present invention. Figure 7 As shown in the figure, the three-dimensional registration of the real foot can be achieved by the following steps:

[0102] S1, collects multi-view image data of the real foot to obtain the local color image and the depth information corresponding to the pixels.

[0103] S2 converts the depth information of the multi-view image into a three-dimensional point cloud. At this time, each piece of the three-dimensional point cloud is located in the camera coordinate system of the corresponding image. Then, a point cloud pose graph is established during the acquisition process, and each point cloud constitutes a node in the pose graph.

[0104] S3, perform serialized registration. If there is enough overlap between point clouds, add connecting edges to the point clouds. Connecting edges are divided into adjacent edges and loop edges. Adjacent edges connect two nodes Pi and Pi+1 that are adjacent in time sequence. Loop edges connect two nodes Pi and Pj that are adjacent in time sequence. There is no requirement for adjacent time sequence. When the overlapping area meets a certain threshold, it is added. When establishing the pose graph, it is necessary to solve the weight values ​​of all connecting edges and the corresponding transformation matrix. This solution is achieved through the ICP point cloud registration algorithm, which is also a serialized registration process.

[0105] S4, perform global registration based on the pose graph, first define the energy function as follows:

[0106]

[0107] Among them, {Pk} is the quantity to be solved - the pose of each point cloud in the global coordinate system, {xk} is the observed point cloud data, and ek(·) is the cost function defined between the observed quantity and the quantity to be solved. The LM nonlinear optimization method is used to solve the energy function, and finally the pose of each point cloud in the global coordinate system is obtained. As an optional embodiment, the coordinate system of the point cloud P0 at the start time of recording t=0 can be used as the global coordinate system. After global alignment and solution, the 6DoF pose of each point cloud in the point cloud P0 coordinate system is obtained. After transformation according to the pose result, the point clouds at all times are aligned to the point cloud P0 coordinate system, and the reconstructed foot model is obtained after fusion.

[0108] As an alternative implementation, when reconstructing a real foot in 3D, the global registration process can also employ sequential registration in time. This involves registering two adjacent frames of the acquired point cloud {x0, x1, x2, ..., xi, xi+1, ..., xn}, and then transferring the pose of each point cloud in the global coordinate system (x0) through a transformation. However, this method is prone to significant drift errors due to the accumulated errors during the transfer process, resulting in significant registration errors between x0 and xn.

[0109] Figure 8 is a schematic diagram of aligning a virtual shoe, a virtual foot, and a reconstructed foot according to an optional embodiment of the present invention. Figure 8As shown in the figure, the alignment of the virtual shoe, virtual foot and reconstructed foot can be achieved as follows: first, the relationship between the virtual shoe and the virtual foot is predefined; then, the virtual foot and the reconstructed foot are aligned. The alignment process can be based on the coarse point cloud registration of FPFH and the fine point cloud registration of point-to-surface ICP to obtain the transformation matrix, and then the virtual foot and the reconstructed foot are aligned based on the transformation matrix; finally, based on the alignment relationship between the virtual shoe and the virtual foot, the posture of the virtual shoe is adjusted to obtain a virtual shoe aligned with the virtual foot. Since the virtual foot has been aligned with the reconstructed foot, the virtual shoe obtained at this time is also aligned with the reconstructed foot.

[0110] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0111] Through the description of the above embodiments, those skilled in the art can clearly understand that the data processing method according to the above embodiments can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

[0112] Example 2

[0113] According to an embodiment of the present invention, a data processing device for implementing the above-mentioned data processing method 1 is also provided. Figure 9 : is a structural block diagram of a data processing device according to embodiment 2 of the present invention. Figure 9 As shown, the device includes: a first acquisition module 92, a first processing module 94, a first determination module 96 and a second processing module 98. The data processing device 1 is described in detail below.

[0114] A first acquisition module 92 is used to acquire a color image and a depth image of a physical target object;

[0115] A first processing module 94 is connected to the first acquisition module 92 and is used to fuse the depth image to obtain a three-dimensional reconstructed model of the physical target object;

[0116] a first determining module 96 connected to the first processing module 94 and configured to determine a first pose of the virtual target object in a predetermined coordinate system and a second pose of the virtual object in a predetermined coordinate system based on the three-dimensional reconstructed model;

[0117] The second processing module 98 is connected to the above-mentioned first determination module 96, and is used to project the 3D key points of the virtual target object onto the image plane to obtain 2D key points according to the first posture, and to draw an interactive image of the virtual object and the physical target object in the image plane according to the second posture and color image.

[0118] It should be noted that the first acquisition module 92, the first processing module 94, the first determination module 96, and the second processing module 98 correspond to steps S202 to S208 in Example 1. The examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0119] Example 3

[0120] According to an embodiment of the present invention, a data processing device for implementing the above-mentioned data processing method 2 is also provided. Figure 10 : is a structural block diagram of a data processing device 2 according to embodiment 3 of the present invention. Figure 10 As shown, the device includes: a receiving module 1002 and a detecting module 1004. The second data processing device is described in detail below.

[0121] A receiving module 1002 is configured to receive an input image, wherein the input image includes a physical target object;

[0122] The detection module 1004 is connected to the above-mentioned receiving module 1002 and is used to use a key point detection model to detect the 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the virtual object and the 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second pose of the virtual target object in a predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to the three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object.

[0123] It should be noted that the receiving module 1002 and the detecting module 1004 correspond to steps S302 to S304 in Example 1. The examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0124] Example 4

[0125] According to an embodiment of the present invention, a data processing device for implementing the above-mentioned data processing method 3 is also provided. Figure 11 : is a structural block diagram of a data processing device 3 according to embodiment 4 of the present invention. Figure 11 As shown, the device includes: an acquisition module 1102 and a training module 1104. The data processing device 3 is described in detail below.

[0126] An acquisition module 1102 is configured to acquire multiple sets of training data, wherein the data in the multiple sets of training data include: an interaction image between a physical target object and a virtual object and 2D key points in the interaction image, wherein the 2D key points are obtained by projecting 3D key points of the virtual target object onto an image plane based on a first pose of the virtual target object in a predetermined coordinate system, and the interaction image is obtained by rendering the virtual object on the image plane based on a second pose of the virtual object in the predetermined coordinate system and a collected color image of the physical target object, wherein the first pose and the second pose are determined based on a 3D reconstruction model, and the 3D reconstruction model is obtained by fusing the collected depth image of the physical target object;

[0127] The training module 1104 is connected to the acquisition module 1102 and is used to perform machine training using multiple sets of training data to obtain a key point detection model.

[0128] It should be noted that the acquisition module 1102 and the training module 1104 correspond to steps S402 to S404 in Example 1. The examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0129] Example 5

[0130] According to an embodiment of the present invention, a data processing device for implementing the fourth data processing method is also provided. Figure 12 : is a structural block diagram of a data processing device 4 according to embodiment 5 of the present invention. Figure 12As shown, the device includes: a second acquisition module 1202, a third processing module 1204, a second determination module 1206 and a fourth processing module 1208. The data processing device 4 is described in detail below.

[0131] The second acquisition module 1202 is used to acquire a color image and a depth image of the foot;

[0132] A third processing module 1204 is connected to the second acquisition module 1202 and is used to determine a three-dimensional reconstructed foot model of the foot based on the depth image;

[0133] a second determining module 1206 , connected to the third processing module 1204 , configured to determine a first pose of the virtual foot in a predetermined coordinate system and a second pose of the virtual shoe in a predetermined coordinate system based on the three-dimensional reconstructed foot model;

[0134] The fourth processing module 1208 is connected to the above-mentioned second determination module 1206, and is used to project the 3D foot key points of the virtual foot onto the image plane to obtain 2D foot key points according to the first posture, and to draw a fitting image of the virtual shoe on the image plane according to the second posture and the color image.

[0135] It should be noted that the second acquisition module 1202, the third processing module 1204, the second determination module 1206, and the fourth processing module 1208 correspond to steps S502 to S508 in Example 1. The examples and application scenarios implemented by the multiple modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0136] Example 6

[0137] The embodiment of the present invention can provide a computer terminal, which can be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal can also be replaced by a terminal device such as a mobile terminal.

[0138] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0139] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the data processing method of the application: collecting the color image and depth image of the physical target object; fusing the multiple depth images to obtain a three-dimensional reconstruction model of the physical target object; determining the first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstruction model, and determining the second pose of the virtual object in the predetermined coordinate system; projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points based on the first pose, and drawing the interactive image of the virtual object on the image plane based on the second pose and the color image.

[0140] Optionally, Figure 13 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 13 As shown, the computer terminal may include: one or more (only one is shown in the figure) processors 1302, memory 1304, etc.

[0141] Among them, the memory 1304 can be used to store software programs and modules, such as program instructions / modules corresponding to the data processing method and device in the embodiments of the present invention. The processor 1302 executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the above-mentioned data processing method. The memory 1304 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1304 may further include a memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0142] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: collecting the color image and depth image of the physical target object; fusing the multiple depth images to obtain a three-dimensional reconstruction model of the physical target object; determining the first pose of the virtual target object in a predetermined coordinate system and the second pose of the virtual object in the predetermined coordinate system based on the three-dimensional reconstruction model; projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points based on the first pose, and drawing the interactive image of the virtual object on the image plane based on the second pose and the color image.

[0143] Optionally, the processor may also execute the program code of the following steps: in the case of multiple depth images, a three-dimensional reconstruction model of the physical target object is obtained by fusing the multiple depth images, including: converting the multiple depth images into multiple three-dimensional point clouds; establishing a point cloud pose graph based on the multiple three-dimensional point clouds, wherein each point cloud in the multiple three-dimensional point clouds is a node in the point cloud pose graph; and registering the point cloud pose graph to obtain a three-dimensional reconstruction model of the physical target object.

[0144] Optionally, the processor may also execute the program code of the following steps: registering the point cloud pose graph to obtain a three-dimensional reconstructed model of the physical target object, including: using the iterative closest point ICP method to serialize the point cloud pose graph to obtain a serialized registration result graph of the point cloud pose graph; performing global registration on the serialized registration result graph to obtain a three-dimensional reconstructed model of the physical target object, wherein the pose of each point cloud in the three-dimensional reconstructed model is the pose in the global coordinate system.

[0145] Optionally, the processor may also execute the program code of the following steps: determining the first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstructed model, including: obtaining the third pose of the three-dimensional reconstructed model in the predetermined coordinate system; determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model; and determining the first pose of the virtual target object in the predetermined coordinate system based on the third pose and the transformation matrix.

[0146] Optionally, the processor may also execute the program code of the following steps: determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model, including: aligning the virtual target object with the three-dimensional reconstructed model based on a point cloud coarse registration method of the fast point feature histogram FPFH and a point cloud fine registration method of the point-to-surface ICP to obtain alignment parameters; and determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model based on the alignment parameters.

[0147] Optionally, the processor may also execute the program code of the following steps: determining a second posture of the virtual object in a predetermined coordinate system, including: obtaining an alignment relationship between the virtual object and the virtual target object; and determining the second posture of the virtual object in the predetermined coordinate system based on the first posture and the alignment relationship.

[0148] Optionally, the processor may also execute the program code of the following steps: according to the first posture, projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points, and drawing the interaction image of the virtual object and the physical target object in the image plane according to the second posture and the color image, including: according to the first posture and the camera imaging method, projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points; according to the second posture and the color image, calling a predetermined rendering engine to draw the interaction results between the virtual object and the physical target object in the image plane, and obtaining the interaction image of the virtual object and the physical target object in the image plane.

[0149] Optionally, the processor may also execute the program code of the following steps: based on the second posture and color image, calling a predetermined rendering engine to draw the interaction effect of the virtual object in the image plane to obtain an interaction image of the virtual object in the image plane, including: when calling a predetermined rendering engine to draw the interaction result of the virtual object with the physical target object in the image plane based on the second posture and color image, obtaining multiple interaction images of the virtual object with the physical target object in the image plane by replacing the background and / or adjusting the light source.

[0150] Optionally, the processor may also execute the program code of the following steps: when the acquired color images and depth images are multiple color images and multiple depth images of the physical target object in different states and / or at different angles, 2D key points of the physical target object in different states and / or at different angles and interaction images with virtual objects are obtained respectively.

[0151] Optionally, the processor may also execute the program code of the following steps: generating training data based on 2D key points and interactive images; obtaining different training data corresponding to a variety of physical target objects; and performing machine training using different training data corresponding to a variety of physical target objects to obtain a key point detection model.

[0152] Optionally, the processor may also execute program code of the following steps: the physical target object and the virtual object include any one of the following combinations: feet and virtual shoes; head and virtual hat; wrist and virtual bracelet; human body and virtual clothing; face and virtual facial ornaments.

[0153] Optionally, the processor may also execute the program code of the following steps: receiving an input image, wherein the input image includes a physical target object; using a key point detection model to detect 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second pose of the virtual object in the predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to the three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object.

[0154] Optionally, the processor may also execute the program code of the following steps: obtaining multiple sets of training data, wherein the data in the multiple sets of training data include: interaction images of the physical target object and the virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first posture of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second posture of the virtual object in the predetermined coordinate system and the collected color image of the physical target object, the first posture and the second posture are determined according to the three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object; using multiple sets of training data for machine training to obtain a key point detection model.

[0155] Optionally, the processor may also execute the program code of the following steps: acquiring a color image and a depth image of the foot; determining a three-dimensional reconstructed foot model of the foot based on the depth image; determining a first pose of the virtual foot in a predetermined coordinate system and a second pose of the virtual shoe in a predetermined coordinate system based on the three-dimensional reconstructed foot model; projecting the 3D foot key points of the virtual foot onto the image plane to obtain 2D foot key points based on the first pose, and drawing a fitting image of the virtual shoe on the image plane based on the second pose and the color image.

[0156] It can be understood by those skilled in the art that Figure 13 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 13 It does not limit the structure of the above electronic device. For example, the computer terminal may also include Figure 13More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 13 Different configurations shown.

[0157] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0158] Example 7

[0159] The embodiment of the present invention further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data processing method provided in the first embodiment.

[0160] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0161] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a color image and a depth image of a physical target object; fusing the depth image to obtain a three-dimensional reconstructed model of the physical target object; determining a first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstructed model, and determining a second pose of the virtual object in the predetermined coordinate system; projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points based on the first pose, and drawing an interactive image of the virtual object and the physical target object in the image plane based on the second pose and the color image.

[0162] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: in the case of multiple depth images, fusing the multiple depth images to obtain a three-dimensional reconstructed model of the physical target object, including: converting the multiple depth images into multiple three-dimensional point clouds; establishing a point cloud pose graph based on the multiple three-dimensional point clouds, wherein each point cloud in the multiple three-dimensional point clouds is a node in the point cloud pose graph; and aligning the point cloud pose graph to obtain a three-dimensional reconstructed model of the physical target object.

[0163] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: registering the point cloud pose graph to obtain a three-dimensional reconstructed model of the physical target object, including: using an iterative closest point ICP method to serialize and register the point cloud pose graph to obtain a serialized registration result graph of the point cloud pose graph; performing global registration on the serialized registration result graph to obtain a three-dimensional reconstructed model of the physical target object, wherein the pose of each point cloud in the three-dimensional reconstructed model is the pose in the global coordinate system.

[0164] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: determining the first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstructed model, including: obtaining the third pose of the three-dimensional reconstructed model in the predetermined coordinate system; determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model; and determining the first pose of the virtual target object in the predetermined coordinate system based on the third pose and the transformation matrix.

[0165] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model, including: a point cloud coarse registration method based on the fast point feature histogram FPFH and a point cloud fine registration method based on the point-to-surface ICP, aligning the virtual target object with the three-dimensional reconstructed model to obtain alignment parameters; and determining the transformation matrix of the virtual target object to the three-dimensional reconstructed model based on the alignment parameters.

[0166] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: determining a second posture of the virtual object in a predetermined coordinate system, including: obtaining an alignment relationship between the virtual object and the virtual target object; and determining the second posture of the virtual object in the predetermined coordinate system based on the first posture and the alignment relationship.

[0167] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points according to the first posture, and drawing the interaction image of the virtual object and the physical target object in the image plane according to the second posture, including: projecting the 3D key points of the virtual target object onto the image plane to obtain 2D key points according to the first posture and the camera imaging method; calling a predetermined rendering engine to draw the interaction results between the virtual object and the physical target object in the image plane according to the second posture and the color image, to obtain the interaction image of the virtual object and the physical target object in the image plane.

[0168] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: calling a predetermined rendering engine to draw the interaction result of the virtual object with the physical target object in the image plane according to the second posture and color image, and obtaining an interaction image of the virtual object with the physical target object in the image plane, including: when calling a predetermined rendering engine to draw the interaction result of the virtual object with the physical target object in the image plane according to the second posture and color image, obtaining multiple interaction images of the virtual object with the physical target object in the image plane by replacing the background and / or adjusting the light source.

[0169] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: when the acquired color images and depth images are color images and multiple depth images of the physical target object in different states and / or at different angles, 2D key points of the physical target object in different states and / or at different angles and images after interaction with virtual objects are obtained respectively.

[0170] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: generating training data based on 2D key points and interactive images; obtaining different training data corresponding to a variety of physical target objects; and performing machine training using different training data corresponding to a variety of physical target objects to obtain a key point detection model.

[0171] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the physical target object and the virtual object include any one of the following combinations: feet and virtual shoes; head and virtual hat; wrist and virtual bracelet; human body and virtual clothing; face and virtual facial ornaments.

[0172] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving an input image, wherein the input image includes a physical target object; using a key point detection model to detect 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second pose of the virtual object in the predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to the three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object.

[0173] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining multiple sets of training data, wherein the data in the multiple sets of training data include: interaction images of the physical target object and the virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting the 3D key points of the virtual target object onto the image plane according to the first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to the second pose of the virtual object in the predetermined coordinate system and the collected color image of the physical target object, the first pose and the second pose are determined according to the three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object; using multiple sets of training data for machine training to obtain a key point detection model.

[0174] Optionally, in this embodiment, the storage medium is configured to store program code for executing the following steps: acquiring a color image and a depth image of the foot; determining a three-dimensional reconstructed foot model of the foot based on the depth image; determining a first pose of the virtual foot in a predetermined coordinate system based on the three-dimensional reconstructed foot model, and determining a second pose of the virtual shoe in a predetermined coordinate system; projecting the 3D foot key points of the virtual foot onto the image plane to obtain 2D foot key points based on the first pose, and drawing a fitting image of the virtual shoe on the image plane based on the second pose and the color image.

[0175] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0176] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0178] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0181] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: include: Collect color images and depth images of physical target objects; determining a three-dimensional reconstructed model of the physical target object according to the depth image; Determining a first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstructed model, and determining a second pose of the virtual object in the predetermined coordinate system; projecting 3D key points of the virtual target object onto an image plane to obtain 2D key points according to the first pose, and rendering an interaction image of the virtual object and the physical target object in the image plane according to the second pose and the color image; Determining the second posture of the virtual object in the predetermined coordinate system includes: obtaining an alignment relationship between the virtual object and the virtual target object; and determining the second posture of the virtual object in the predetermined coordinate system based on the first posture and the alignment relationship.

2. The method according to claim 1, characterized in that In the case where there are multiple depth images, determining a three-dimensional reconstructed model of the physical target object according to the multiple depth images includes: Converting the multiple depth images into multiple three-dimensional point clouds; Establishing a point cloud pose graph based on the multiple three-dimensional point clouds, wherein each point cloud in the multiple three-dimensional point clouds is a node in the point cloud pose graph; The point cloud pose graph is registered to obtain a three-dimensional reconstructed model of the physical target object.

3. The method according to claim 2, characterized in that Registering the point cloud pose graph to obtain a three-dimensional reconstructed model of the physical target object includes: Using an iterative closest point (ICP) method, the point cloud pose graph is serialized and registered to obtain a serialized registration result graph of the point cloud pose graph; Global registration is performed on the serialized registration result images to obtain a three-dimensional reconstructed model of the physical target object, wherein the pose of each point cloud in the three-dimensional reconstructed model is a pose in a global coordinate system.

4. The method according to claim 1, wherein Determining a first pose of the virtual target object in a predetermined coordinate system according to the three-dimensional reconstructed model includes: Acquire a third pose of the three-dimensional reconstructed model in the predetermined coordinate system; Determining a transformation matrix for transforming the virtual target object into the three-dimensional reconstructed model; The first posture of the virtual target object in the predetermined coordinate system is determined according to the third posture and the transformation matrix.

5. The method according to claim 4, characterized in that Determining a transformation matrix for transforming the virtual target object to the three-dimensional reconstructed model includes: Based on a point cloud coarse registration method of a fast point feature histogram (FPFH) and a point cloud fine registration method of a point-to-surface ICP, the virtual target object is aligned with the three-dimensional reconstructed model to obtain alignment parameters; A transformation matrix for transforming the virtual target object to the three-dimensional reconstructed model is determined according to the alignment parameters.

6. The method according to claim 1, characterized in that The method further comprises: projecting 3D key points of the virtual target object onto an image plane to obtain 2D key points according to the first pose; and drawing an interaction image of the virtual object and the physical target object in the image plane according to the second pose and the color image, including: According to the first posture and the camera imaging method, projecting the 3D key points of the virtual target object onto an image plane to obtain 2D key points; According to the second posture and the color image, a predetermined rendering engine is called to draw the interaction result between the virtual object and the physical target object in the image plane to obtain an interaction image between the virtual object and the physical target object in the image plane.

7. The method according to claim 6, characterized in that Invoking a predetermined rendering engine to draw an interaction result between the virtual object and the physical target object in the image plane according to the second posture and the color image, to obtain an interaction image between the virtual object and the physical target object in the image plane, including: When a predetermined rendering engine is called to draw the interaction result of the virtual object with the physical target object in the image plane based on the second posture and the color image, multiple interaction images of the virtual object with the physical target object in the image plane are obtained by replacing the background and / or adjusting the light source.

8. The method according to any one of claims 1 to 7, characterized in that When the collected color image and the collected depth image are multiple color images and multiple depth images of the physical target object in different states and / or at different angles, the 2D key points of the physical target object in different states and / or at different angles and the interaction images with the virtual object are obtained respectively.

9. The method according to claim 8, characterized in that Also includes: Generating training data according to the 2D key points and the interaction image; Acquiring different training data corresponding to a plurality of physical target objects; The machine training is performed using different training data corresponding to a variety of physical target objects to obtain a key point detection model.

10. The method according to claim 9, characterized in that The physical target object and the virtual object include any one of the following combinations: feet and virtual shoes; head and virtual hat; wrist and virtual bracelet; human bodies and virtual costumes; Facial and virtual facial accessories.

11. A data processing method, characterized in that: include: receiving an input image, wherein the input image includes a physical target object; A key point detection model is used to detect 2D key points of the physical target object in the input image, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the virtual object and 2D key points in the interaction image, the 2D key points are obtained by projecting 3D key points of the virtual target object onto an image plane according to a first pose of the virtual target object in a predetermined coordinate system, the interaction image is obtained by drawing the virtual object on the image plane according to a second pose of the virtual target object in the predetermined coordinate system and a collected color image of the physical target object, the first pose and the second pose are determined according to a three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the collected depth image of the physical target object; The second posture is determined according to the three-dimensional reconstructed model, including: obtaining the alignment relationship between the virtual object and the virtual target object; and determining the second posture of the virtual object in the predetermined coordinate system according to the first posture and the alignment relationship.

12. A data processing method, characterized in that: include: Acquire multiple sets of training data, wherein the data in the multiple sets of training data include: an interaction image between a physical target object and a virtual object and 2D key points in the interaction image, the 2D key points being obtained by projecting 3D key points of the virtual target object onto an image plane based on a first pose of the virtual target object in a predetermined coordinate system, the interaction image being obtained by rendering the virtual object on the image plane based on a second pose of the virtual object in the predetermined coordinate system and a captured color image of the physical target object, the first pose and the second pose being determined based on a three-dimensional reconstruction model, and the three-dimensional reconstruction model being obtained by fusing a captured depth image of the physical target object; Performing machine training using the multiple sets of training data to obtain a key point detection model; The second posture is determined according to the three-dimensional reconstructed model, including: obtaining the alignment relationship between the virtual object and the virtual target object; and determining the second posture of the virtual object in the predetermined coordinate system according to the first posture and the alignment relationship.

13. A data processing device, characterized in that: include: An acquisition module, used to acquire color images and depth images of physical target objects; A first processing module is configured to fuse the multiple depth images to obtain a three-dimensional reconstructed model of the physical target object; a determination module, configured to determine a first pose of the virtual target object in a predetermined coordinate system based on the three-dimensional reconstructed model, and to determine a second pose of the virtual object in the predetermined coordinate system; a second processing module, configured to project the 3D key points of the virtual target object onto an image plane to obtain 2D key points according to the first posture, and to draw an interaction image of the virtual object and the physical target object in the image plane according to the second posture and the color image; The determining module is further configured to obtain an alignment relationship between the virtual object and the virtual target object; and determine the second posture of the virtual object in the predetermined coordinate system according to the first posture and the alignment relationship.

14. A data processing device, characterized in that: include: A receiving module, configured to receive an input image, wherein the input image includes a physical target object; a detection module, configured to detect 2D key points of the physical target object in the input image using a key point detection model, wherein the key point detection model is obtained through machine training using multiple sets of training data, and the data in the multiple sets of training data include: an interaction image between the physical target object and the virtual object and 2D key points in the interaction image, the 2D key points being obtained by projecting 3D key points of the virtual target object onto an image plane based on a first pose of the virtual target object in a predetermined coordinate system, the interaction image being obtained by rendering the virtual target object on the image plane based on a second pose of the virtual target object in the predetermined coordinate system and a collected color image of the physical target object, the first pose and the second pose being determined based on a three-dimensional reconstruction model, and the three-dimensional reconstruction model being obtained by fusing a collected depth image of the physical target object; The device is further configured to obtain an alignment relationship between the virtual object and the virtual target object; and determine the second posture of the virtual object in the predetermined coordinate system based on the first posture and the alignment relationship.

15. A data processing device, characterized in that: include: an acquisition module configured to acquire multiple sets of training data, wherein the data in the multiple sets of training data include: an interaction image between a physical target object and a virtual object and 2D key points in the interaction image, wherein the 2D key points are obtained by projecting 3D key points of the virtual target object onto an image plane based on a first pose of the virtual target object in a predetermined coordinate system, and the interaction image is obtained by rendering the virtual object on the image plane based on a second pose of the virtual object in the predetermined coordinate system and a captured color image of the physical target object, wherein the first pose and the second pose are determined based on a three-dimensional reconstruction model, and the three-dimensional reconstruction model is obtained by fusing the captured depth image of the physical target object; A training module, configured to perform machine training using the plurality of sets of training data to obtain a key point detection model; The device is further configured to obtain an alignment relationship between the virtual object and the virtual target object; and determine the second posture of the virtual object in the predetermined coordinate system based on the first posture and the alignment relationship.

16. A storage medium, characterized in that The storage medium includes a stored program, wherein when the program is run, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 12.

17. A computer device, characterized in that: include: memory and processor, The memory stores a computer program; The processor is configured to execute a computer program stored in the memory, and when the computer program is run, the processor is enabled to execute the data processing method according to any one of claims 1 to 12.

18. A data processing method, characterized in that: include: Collect color images and depth images of the foot; determining a three-dimensional reconstructed foot model of the foot according to the depth image; Determining a first pose of the virtual foot in a predetermined coordinate system based on the three-dimensional reconstructed foot model, and determining a second pose of the virtual shoe in the predetermined coordinate system; According to the first posture, projecting the 3D foot key points of the virtual foot onto an image plane to obtain 2D foot key points, and drawing a fitting image of the virtual shoe on the image plane according to the second posture and the color image; Determining the second posture of the virtual shoe in the predetermined coordinate system includes: obtaining an alignment relationship between the virtual shoe and the virtual foot; and determining the second posture of the virtual shoe in the predetermined coordinate system based on the first posture and the alignment relationship.

Images