Virtual fitting system and virtual fitting method

Abstract

This application provides a virtual try-on system and a virtual try-on method, relating to the field of virtual try-on technology. The virtual try-on system includes a virtual try-on helmet and a display device. The virtual try-on helmet includes a helmet body and a first acquisition component and a second acquisition component mounted on the helmet body. The first and second acquisition components unfold in opposite directions when the user wears the virtual try-on system. The first acquisition component is used to acquire first initial data within a first viewpoint, and the second acquisition component is used to acquire second initial data within a second viewpoint. The helmet body is used to determine a target human body point cloud model corresponding to the user based on the first and second initial data. The helmet body is also used to combine a target clothing model with the target human body point cloud model. The display device is used to display the target human body point cloud model and the target clothing model. This application achieves virtual try-on by utilizing the acquisition components on the helmet to monitor human body data.

Description

Technical Field

[0001] This application belongs to the field of virtual fitting, and particularly relates to a virtual fitting system and a virtual fitting method. Background Technology

[0002] Currently, the clothing market still largely relies on traditional fitting methods, which require customers to manually retrieve clothing, try it on (actually wearing the clothes), return it, and have store staff tidy it up. This method is not only inefficient but also frequently results in unnecessary damage to clothing due to customer carelessness. Therefore, virtual fitting in smart scenarios has emerged and flourished.

[0003] However, in the existing technology, when designing virtual try-on, the usual methods are to use a ring rail to drive the sensor to rotate, a vertical rail to drive the sensor to rotate, or a turntable to drive the person to rotate. These methods are costly, structurally complex, and not conducive to the application and implementation of the invention. Therefore, there is an urgent need for a virtual try-on method that is simple in structure, easy to apply and implement. Summary of the Invention

[0004] This application provides a virtual fitting system and a virtual fitting method. It designs a virtual fitting helmet carrying a data acquisition component, which monitors human body data by using the data acquisition component on the helmet to achieve virtual fitting.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] In a first aspect, a virtual fitting system is provided, comprising: a virtual fitting helmet and a display device; the virtual fitting helmet includes: a helmet body, and a first acquisition component and a second acquisition component mounted on the helmet body; the first acquisition component and the second acquisition component unfold in opposite directions when the user wears the virtual fitting system, the first acquisition component is used to acquire first initial data within a first viewpoint, and the second acquisition component is used to acquire second initial data within a second viewpoint, the first viewpoint and the second viewpoint being opposite to each other and both facing the user; the helmet body is used to determine a target human body point cloud model corresponding to the user based on the first initial data and the second initial data; the helmet body is also used to combine a target clothing model with the target human body point cloud model; the display device is used to display the combined target human body point cloud model and the target clothing model.

[0007] It should be understood that the target clothing model is one or more clothing models, and the clothing model selected by the user for display.

[0008] The virtual try-on system provided in the first aspect collects human body data from the user's perspective using two sets of acquisition components on the helmet itself. This allows for the creation of a relatively complete human body point cloud model. The user then simply selects a clothing model for display, making the operation simple. Because the collected data has a comprehensive coverage, the resulting human body point cloud model is more accurate, leading to better results in matching clothing models.

[0009] In one possible implementation of the first aspect, the first acquisition component includes: a first lidar sensor and a first camera; the second acquisition component includes: a second lidar sensor and a second camera; the first lidar sensor is used to acquire first point cloud data, the first camera is used to acquire first image data, and the first initial data includes the first point cloud data and the first image data; the second lidar sensor is used to acquire second point cloud data, the second camera is used to acquire second image data, and the second initial data includes the second point cloud data and the second image data.

[0010] In this implementation, the point cloud data collected by the lidar sensor and the image data collected by the camera are combined, resulting in more and richer data. The subsequent human body model and clothing model are then presented on the human body model with better performance.

[0011] In one possible implementation of the first aspect, the helmet body is further configured to: determine an initial human point cloud model based on the first point cloud data and the second point cloud data using a human body modeling algorithm, and use the initial human point cloud model as the target human point cloud model; wherein the human body modeling algorithm includes at least one of denoising, stitching, and registration.

[0012] In this implementation, an initial human point cloud model can be established based on the first initial data and the second initial data.

[0013] In one possible implementation of the first aspect, the helmet body is further used to: repair the initial human point cloud model using a side repair algorithm to obtain the target human point cloud model.

[0014] In this implementation, since the initial human point cloud model is not perfect, it can be repaired using a side-mounted repair algorithm to obtain a more complete model.

[0015] In one possible implementation of the first aspect, the helmet body is further configured to: correct the first three-dimensional spatial coordinates corresponding to the first lidar sensor based on the first point cloud data, and correct the coordinate values ​​of the first point cloud data using the corrected first three-dimensional spatial coordinates; the helmet body is further configured to: correct the second three-dimensional spatial coordinates corresponding to the second lidar sensor based on the second point cloud data, and correct the coordinate values ​​of the second point cloud data using the corrected second three-dimensional spatial coordinates.

[0016] In this implementation, by correcting the three-dimensional spatial coordinates, the coordinate values ​​of the point cloud data can be further corrected, thereby reducing errors caused by external factors such as movement and improving the accuracy of the collected data.

[0017] In one possible implementation of the first aspect, the helmet body is further configured to: segment the first point cloud data using a point cloud segmentation algorithm to generate first human point cloud data and first background point cloud data; determine first ground point cloud data from the first background point cloud data using a ground-finding algorithm; and correct the first three-dimensional spatial coordinates corresponding to the first lidar sensor based on the first ground point cloud data; the helmet body is further configured to: segment the second point cloud data using the point cloud segmentation algorithm to generate second human point cloud data and second background point cloud data; determine second ground point cloud data from the second background point cloud data using the ground-finding algorithm; and correct the second three-dimensional spatial coordinates corresponding to the second lidar sensor based on the second ground point cloud data.

[0018] In this implementation, the accuracy of the Z-axis in the determined three-dimensional spatial coordinates can be improved by using point cloud segmentation algorithms and ground-finding algorithms, which in turn can improve the accuracy of the determined three-dimensional spatial coordinates, and further improve the accuracy of the coordinate values ​​of the corrected first point cloud data and second point cloud data.

[0019] In one possible implementation of the first aspect, the helmet body is further used to: use a matching algorithm to determine one or more clothing models that match the target human body point cloud model.

[0020] In this implementation, a matching algorithm can be used to determine the clothing model that matches the user's human body point cloud model, narrowing the range of selectable clothing models and allowing the user to try on clothes only from the matched clothing models, thus improving the efficiency of virtual try-on.

[0021] In one possible implementation of the first aspect, the helmet body is further configured to: determine the target human posture category and coordinate values ​​of key human body parts corresponding to the user based on the first initial data and the second initial data using a human posture recognition algorithm, wherein the human posture category includes at least one of standing, squatting, and sitting postures, and the target human posture category is one of the human posture categories; adjust the shape of the target clothing model based on the target human posture category and the coordinate values ​​of the key human body parts; and the display device is further configured to display the adjusted target clothing model.

[0022] In this implementation, by determining the target human body posture category and the coordinate values ​​of key human body parts, the target clothing model can be adjusted to make its display effect more natural and more consistent with the target human body point cloud model. Users can try on the clothes through the display device, experience the effect of the clothes on the body, and also change their human body posture to see the corresponding display effect of the clothes in different postures. Based on the display effect, users can easily and conveniently determine their favorite clothing without physically trying it on.

[0023] Secondly, a virtual fitting method is provided, which can be applied to the virtual fitting system in the first aspect or any possible implementation thereof.

[0024] The method comprises:

[0025] When a user wears the virtual fitting system, the first acquisition component acquires first initial data from a first perspective, and the second acquisition component acquires second initial data from a second perspective. The first and second perspectives are opposite each other and both face the user. Based on the first and second initial data, a target human body point cloud model corresponding to the user is determined. The target clothing model is combined with the target human body point cloud model and displayed on the display device. The target clothing model is one or more clothing models, and the clothing model selected by the user for display is displayed.

[0026] The second method provides a virtual try-on approach. It utilizes two sets of data acquisition components on the helmet to collect human body data from the user's perspective, thereby creating a relatively complete human point cloud model. The user then simply selects a clothing model for display, making the process simple. Because the collected data has a comprehensive coverage, the resulting human point cloud model is more accurate, leading to better matching with clothing models.

[0027] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the relevant processing steps of the virtual try-on method as described in the second aspect above.

[0028] Fourthly, a computer-readable storage medium is provided, wherein a computer program or instructions are stored therein, and when a computer reads and executes the computer program or instructions, the computer performs the virtual try-on method as described in the second aspect above.

[0029] Fifthly, a computer program product is provided that, when run on an electronic device, causes the electronic device to perform the virtual try-on method as described in the second aspect above.

[0030] This application provides a virtual try-on system and method. By setting two sets of acquisition components on the helmet body, human body data is collected when facing the user's viewpoint, thereby establishing a relatively complete human body point cloud model. Then, a matching algorithm is used to determine the clothing model that matches the human body point cloud model, allowing the user to select and display it. The operation is simple. Because the collected data has a comprehensive coverage, the determined human body point cloud model is more accurate, and the subsequent matching of clothing models is more effective. Attached Figure Description

[0031] Figure 1 This is a front view of a virtual fitting system in a non-working state, as provided in an embodiment of this application.

[0032] Figure 2 This is a side view of a virtual fitting system in operation, as provided in an embodiment of this application.

[0033] Figure 3 This is a flowchart illustrating a virtual try-on method provided in an embodiment of this application;

[0034] Figure 4 This is a scene diagram of a user wearing a virtual fitting system provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0036] Figure label:

[0037] 1-Virtual fitting system; 100-Virtual fitting helmet; 110-First acquisition component; 111-First lidar sensor; 112-First camera; 113-First support rod; 120-Second acquisition component; 121-Second lidar sensor; 122-Second camera; 123-Second support rod; 130-Helmet body; 200-Display device. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0039] First, the virtual try-on system provided in the embodiments of this application will be introduced.

[0040] Figure 1 This diagram shows a front view of a virtual try-on system provided in an embodiment of this application in a non-working state. Figure 2 This diagram shows a side view of a virtual try-on system in operation, according to an embodiment of this application.

[0041] like Figure 1 and Figure 2 As shown, the virtual fitting system 1 provided in this application embodiment includes two parts: a virtual fitting helmet 100 and a display device 200.

[0042] The virtual fitting helmet 100 is used to collect and process data, and the display device 200 is used to display the results of the virtual fitting.

[0043] The display device 200 can be installed on the virtual fitting helmet 100. The display device 200 and the virtual fitting helmet 100 can be an integrated structure; alternatively, the display device 200 and the virtual fitting helmet 100 can be separate, meaning they are two products that can be manufactured independently and only interact with each other using communication methods during use.

[0044] Alternatively, the display device 200 and the virtual fitting headset 100 can be detachable, meaning they can be installed together or removed. For example, when the display device 200 is installed on the virtual fitting headset 100, the display device 200 and the virtual fitting headset 100 can be connected via an interface for data transmission and information exchange; however, when the display device 200 is removed from the virtual fitting headset 100, it cannot function.

[0045] The structural relationship between the display device 200 and the virtual fitting helmet 100 can be configured as needed, and this application embodiment does not impose any limitations on this. For example, the display device 200 can be a mobile phone, virtual reality glasses, etc.

[0046] Combination Figure 1 As shown, the virtual fitting helmet 100 includes: a first acquisition component 110, a second acquisition component 120 and a helmet body 130, wherein the first acquisition component 110 and the second acquisition component 120 are both installed on the helmet body 130.

[0047] It should be understood that the first acquisition component 110 and the second acquisition component 120 can be mounted on the helmet body 130 in either a fixed or detachable manner. Of course, other methods are also possible, and the specific design can be tailored to the specific needs. This application embodiment does not impose any limitations on this.

[0048] In some possible implementations, the first acquisition component 110 may include a first lidar sensor 111 and a first camera 112, and the second acquisition component 120 may include a second lidar sensor 121 and a second camera 122.

[0049] In some possible implementations, the first lidar sensor 111 and the second lidar sensor 121 can be area array lidar, multi-line lidar, single-line lidar, or a combination of single-line lidar and gimbal. The specific configuration can be set as needed, and this application embodiment does not impose any restrictions on this.

[0050] It should be understood that in order to use the first lidar sensor 111 and the first camera 112 to photograph a user wearing the virtual fitting helmet 100, the first lidar sensor 111 and the first camera 112 need to be aimed at the user. However, in order to make the scanning range of the first lidar sensor 111 larger, the field of view of the first camera 112 wider, and the data acquired more, there needs to be a certain distance between the first lidar sensor 111 and the first camera 112 and the user on the same horizontal plane.

[0051] In order to ensure that the first lidar sensor 111 and the first camera 112 are at a certain distance from the user on the same horizontal plane, the first acquisition component 110 may further include: a first support rod 113, which is used to set the first lidar sensor 111 and the first camera 112 and to ensure that the first lidar sensor 111 and the first camera 112 are at a certain distance from the user wearing the virtual fitting helmet 100 on the same horizontal plane.

[0052] Based on this, Figure 2 As shown, in order to create a certain distance on the same horizontal plane, if the first end of the first support rod 113 is connected to the helmet body 130, the first lidar sensor 111 and the first camera 112 can be set at the end of the first support rod 113 away from the helmet body 130 (for example, referred to as the second end).

[0053] It should be understood that, in order to prevent the first support rod 113 from causing unnecessary obstruction during shooting and affecting the integrity of the data collected by the first lidar sensor 111 and the first camera 112, the first lidar sensor 111 and the first camera 112 can be set on the side of the first support rod 113 closest to the helmet body 130. In other words, when the user is wearing the helmet, the first lidar sensor 111 and the first camera 112 can be set on the side of the first support rod 113 closest to the user, or the side closer to the ground. This way, the first support rod 113 will not be captured during data collection, reducing the collection of unnecessary data.

[0054] It should also be understood that the first support rod 113 can be telescopic, retracting when not in use and extending when the user is trying on clothes. Figure 2 As shown, the first support rod 113 can be parallel to the -x direction. When the virtual fitting helmet 100 is in working state, it extends along the -x direction; when the virtual fitting helmet 100 is not in working state, it retracts along the x direction. Furthermore, to save even more space, the retracted first support rod 113 can be rotated to the same direction as the second support rod 123, for example, both can be rotated to the x direction, allowing them to be placed side-by-side and occupying less space.

[0055] Similarly, in order to ensure that the second lidar sensor 121 and the second camera 122 are at a certain distance from the user on the same horizontal plane, the second acquisition component 120 may further include: a second support rod 123, which is used to set the second lidar sensor 121 and the second camera 122 and to ensure that the second lidar sensor 121 and the second camera 122 are at a certain distance from the user wearing the virtual fitting helmet 100 on the same horizontal plane.

[0056] Based on this, Figure 2 As shown, in order to create a certain distance on the same horizontal plane, if the first end of the second support rod 123 is connected to the helmet body 130, the second lidar sensor 121 and the second camera 122 can be set at the end of the second support rod 123 away from the helmet body 130 (for example, referred to as the second end).

[0057] To prevent the second support rod 123 from causing unnecessary obstruction during shooting and affecting the integrity of the data collected by the second lidar sensor 121 and the second camera 122, the second lidar sensor 121 and the second camera 122 can be positioned on the side of the second support rod 123 closest to the helmet body 130. In other words, when the user is wearing the helmet, the second lidar sensor 121 and the second camera 122 can be positioned on the side of the second support rod 123 closest to the user, or the side closer to the ground. This way, the second support rod 123 will not be captured during data collection, reducing the collection of unnecessary data.

[0058] It should also be understood that the second support rod 123 can be telescopic, retracting when not in use and extending when the user is trying on clothes. Figure 2 As shown, the second support rod 123 can be parallel to the x-direction. When the virtual fitting helmet 100 is in working state, it extends along the x-direction; when the virtual fitting helmet 100 is not in working state, it retracts along the -x-direction. Furthermore, to save even more space, the retracted second support rod 123 can be rotated to the same direction as the first support rod 113, for example, both can be rotated to the -x-direction, thus allowing them to be placed side-by-side and occupying less space.

[0059] Of course, the first lidar sensor 111 and the first camera 112 can also be located at other positions on the first support rod 113, and the second lidar sensor 121 and the second camera 122 can also be located at other positions on the second support rod 123. The specific settings can be made as needed, and this application embodiment does not impose any restrictions on this.

[0060] The LiDAR-based virtual fitting system 1 provided in this application embodiment is suitable for various scenarios, such as street advertising, temporary use, and store rental. It only requires plugging in a power source and installing the relevant program in the virtual fitting headset to be used. Its application scenarios are very wide-ranging and its usage is very simple. Because the virtual fitting system 1 provided in this application embodiment is installation-free, easy to move, and convenient to rent, it has a low cost and can be quickly implemented.

[0061] The virtual try-on system 1 provided in this application embodiment has been briefly introduced above. The virtual try-on method provided in this application embodiment will now be described in detail with reference to the accompanying drawings. The virtual try-on method provided in this application embodiment is applied to the aforementioned virtual try-on system 1.

[0062] Figure 3 This is a flowchart illustrating a virtual try-on method provided in an embodiment of this application. Figure 4 This is a scene diagram illustrating a user wearing a virtual fitting system, provided as an embodiment of this application. Figure 3 As shown, the method includes the following steps S110 to S200.

[0063] S110. When the virtual fitting helmet 100 is worn on the user's head, the first acquisition component 110 and the second acquisition component 120 installed on the helmet body 130 enter the working state.

[0064] The "operating state" setting indicates that the first acquisition component 110 and the second acquisition component 120 unfold in opposite directions and begin acquiring initial data in the opposite directions. Here, "opposite direction" means that the first acquisition component 110 acquires data in the direction of the second acquisition component 120, and the second acquisition component 120 acquires data in the direction of the first acquisition component 110.

[0065] For example, such as Figure 4 As shown, the first support rod 113 in the first acquisition component 110 extends along the -x direction, and the second support rod 123 in the second acquisition component 120 extends along the x direction. The first acquisition component 110 acquires first initial data in the x direction, and the second acquisition component 120 acquires second initial data in the -x direction.

[0066] It should be understood that sensors can be installed on the helmet body 130. When the sensors detect that a user is wearing the helmet body 130, the virtual fitting system 1 automatically enters the working state. Alternatively, the virtual fitting system 1 can also enter the working state in response to the user's first operation.

[0067] The user's first operation can be used to instruct the user to press, speak, or perform other operations. The specific method can be set as needed, and this application embodiment does not impose any limitations on this. For example, a switch button can be installed on the helmet body 130 of the virtual fitting helmet 100. In response to the user's pressing of the switch button, the first acquisition component 110 and the second acquisition component 120 enter the working state.

[0068] It should be understood that the first acquisition component 110 and the second acquisition component 120 are in a non-operating state, such as a power-off state or a sleep state. For example, in the power-off or sleep state, the first support rod 113 in the first acquisition component 110 is retracted, and the second support rod 123 in the second acquisition component 120 is also retracted. Furthermore, the first acquisition component 110 and the second acquisition component 120 can be rotated to be arranged side-by-side in the same direction. In this case, the first acquisition component 110 and the second acquisition component 120 are essentially folded up, and neither acquires data.

[0069] S120, the first acquisition component 110 acquires the first initial data, and the second acquisition component 120 acquires the second initial data.

[0070] Since the first acquisition component 110 includes a first lidar sensor 111 and a first camera 112, and the second acquisition component 120 includes a second lidar sensor 121 and a second camera 122, the first acquisition component 110 acquires first initial data, including:

[0071] The first lidar sensor 111 acquires first point cloud data, and the first camera 112 acquires first image data. At this time, the first initial data includes the first point cloud data and the first image data.

[0072] The second acquisition component 120 acquires the second initial data, including:

[0073] The second lidar sensor 121 collects second point cloud data, and the second camera 122 collects second image data.

[0074] It should be understood that the first point cloud data, the first image data, the second point cloud data, and the second image data are all data collected in the direction of the user.

[0075] It should be understood that point cloud data refers to recording at least one of the wearer's location information, color information, and reflectance information in the form of points. Image data, for example, records the color information of the wearer and the surrounding environment in the form of three primary colors (red, green, and blue).

[0076] It should be understood that, since the first acquisition component 110 and the second acquisition component 120 are positioned in a straight line above the wearer's head when in operation, the first acquisition component 110 faces the wearer to collect the wearer's initial data, and the second acquisition component 120 also faces the wearer to collect the wearer's second initial data. Therefore, the data collected by the first acquisition component 110 and the second acquisition component 120 are actually facing each other. This allows for the capture of initial data from the user's entire 360° circumference.

[0077] For example, such as Figure 3 As shown, the first lidar sensor 111 and the first camera 112 can be located at the second end of the first acquisition component 110 and face the user to take pictures. The second lidar sensor 121 and the second camera 122 can be located at the second end of the second acquisition component 120 and face the user to take pictures. The first acquisition component 110 is used to acquire environmental data in the front of the user and the surrounding area near the back of the user, and the second acquisition component 120 is used to acquire environmental data in the surrounding area near the back of the user and the front of the user.

[0078] S130, the helmet body 130 corrects the first three-dimensional spatial coordinates corresponding to the first lidar sensor 111 based on the first point cloud data, and uses the corrected first three-dimensional spatial coordinates to correct the coordinate values ​​of the first point cloud data.

[0079] The helmet body 130 corrects the second three-dimensional spatial coordinates corresponding to the second lidar sensor based on the second point cloud data, and uses the corrected second three-dimensional spatial coordinates to correct the coordinate values ​​of the second point cloud data.

[0080] It should be understood that when a user wears the virtual fitting helmet 100, they may shake or turn their head, which could cause the angle of the virtual fitting helmet 100 to deviate, resulting in uncertainty in the reference angles when the first lidar sensor 111 and the second lidar sensor 121 collect data. Therefore, in order to reduce errors and improve the accuracy of the collected data, it is necessary to correct the three-dimensional spatial coordinates of the first lidar sensor 111 and the second lidar sensor 121 in real time.

[0081] The helmet body 130 may include a storage unit and a processing unit. The processing unit can correct the first three-dimensional spatial coordinates corresponding to the first lidar sensor 111 based on the first point cloud data, and can also correct the second three-dimensional spatial coordinates corresponding to the second lidar sensor 121 based on the second point cloud data. The processing unit can also correct the coordinate values ​​of the first point cloud data based on the corrected first three-dimensional spatial coordinates, and correct the coordinate values ​​of the second point cloud data based on the corrected second three-dimensional spatial coordinates. The storage unit can store the corrected coordinate values ​​of the first point cloud data and the corrected coordinate values ​​of the second point cloud data.

[0082] Optionally, a point cloud segmentation method can be used to segment the first point cloud data to generate first human body point cloud data and first background point cloud data; a ground-finding algorithm can be used to determine the first ground point cloud data from the first background point cloud data. Then, based on the first ground point cloud data, the first three-dimensional spatial coordinates of the first lidar sensor 111 are corrected.

[0083] Similarly, point cloud segmentation methods can be used to segment the second point cloud data, generating second human body point cloud data and second background point cloud data. A ground-based search algorithm is then used to determine the second ground point cloud data from the second background point cloud data; based on the second ground point cloud data, the second three-dimensional spatial coordinates corresponding to the second lidar sensor are corrected.

[0084] It should be understood that the ground-finding algorithm can be a plane fitting algorithm or a segmentation algorithm, or of course, a custom algorithm. The specific algorithm can be set as needed, and this application embodiment does not impose any restrictions on it.

[0085] It should be understood that multiple planes can be determined from one frame of first point cloud data, referred to as first planes; the height difference between the first lidar sensor 111 and each first plane is calculated; when the height difference meets the first preset height difference range, the first plane that meets the first preset height difference range is taken as the first candidate plane, thereby determining one or more first candidate planes.

[0086] The first preset height difference range can include many data, such as the first preset height difference range being [1, 2.5] meters.

[0087] Based on one frame of first point cloud data, the point cloud data corresponding to each first candidate plane can be determined. Then, based on multiple frames of first point cloud data, the point cloud data corresponding to each first candidate plane in consecutive frames can be determined. Based on this, the first candidate plane with the smallest change amplitude in consecutive frames is taken as the first ground.

[0088] Using the first ground point cloud data corresponding to the first ground surface, calculate its corresponding normal vector pdt. The orientation of this normal vector is the z-axis of the three-dimensional spatial coordinates corresponding to the first lidar sensor 111, the direction consistent with human vision is the x-axis, and the direction perpendicular to both z and x is the y-axis. Update the determined three-dimensional spatial coordinates to the current first three-dimensional spatial coordinates corresponding to the first lidar sensor 111. Using the current first three-dimensional spatial coordinates, correct the coordinate values ​​of all first point cloud data.

[0089] Similarly, multiple planes can be determined from one frame of second ground point cloud data, referred to as second planes; the height difference between the second lidar sensor 121 and each second plane can be calculated; when the height difference meets the second preset height difference range, the second plane that meets the second preset height difference range is taken as the second candidate plane, thereby determining one or more second candidate planes.

[0090] The second preset height difference range can include a variety of data, such as a second preset height difference range of [1, 2.5] meters.

[0091] Based on one frame of second point cloud data, the point cloud data corresponding to each second candidate plane can be determined. Then, based on multiple frames of second point cloud data, the point cloud data of each second candidate plane across multiple consecutive frames can be determined. Based on this, the second candidate plane with the smallest change amplitude across multiple consecutive frames is selected as the second ground.

[0092] Using the second ground point cloud data corresponding to the second ground, calculate its corresponding normal vector pdt. The orientation of this normal vector is the z-axis of the three-dimensional spatial coordinates corresponding to the second lidar sensor 121, the direction consistent with human vision is the x-axis, and the direction perpendicular to both z and x is the y-axis. Update the determined three-dimensional spatial coordinates to the current second three-dimensional spatial coordinates corresponding to the second lidar sensor 121. Using the current second three-dimensional spatial coordinates, correct the coordinate values ​​of all second point cloud data.

[0093] It should be understood that when the first acquisition component 110 and the second acquisition component 120 acquire multiple cycles in real time, that is, acquire multiple sets of first point cloud data and second point cloud data, the three-dimensional spatial coordinates corresponding to the first point cloud data and the second point cloud data can be determined respectively, so as to perform real-time correction on the three-dimensional spatial coordinates corresponding to the first lidar sensor 111 and the second lidar sensor 121 respectively, and thus the coordinate values ​​of the first point cloud data and the second point cloud data can be corrected in real time.

[0094] S140. Based on the first point cloud data and the second point cloud data, use the human body modeling algorithm to determine the initial human body point cloud model corresponding to the user.

[0095] Optionally, the human body modeling algorithm may include at least one of denoising, stitching, and registration.

[0096] It should be understood that the first and second point cloud data collected usually contain some noise, so noise reduction processing can be performed first.

[0097] When multiple sets of first point cloud data are collected, the current first point cloud data can be registered with a previous set of first point cloud data as a reference.

[0098] In operation, the first acquisition component 110 and the second acquisition component 120 are deployed in opposite directions, and the first point cloud data and the second point cloud data acquired in the opposite direction are equivalent to acquiring 180° data centered on the user. Therefore, in order to build a complete human body model, the first point cloud data and the second point cloud data need to be stitched together to obtain 360° complete point cloud data.

[0099] For example, when a user wears the virtual fitting helmet 100, the extension directions of the first acquisition component 110 and the second acquisition component 120, which are in working condition, are perpendicular to the direction of the user's line of sight. In this case, for example, the first acquisition component 110 is located on the left side of the body, and the second acquisition component 120 is located on the right side. The first acquisition component 110 acquires point cloud data from the left side of the body, and the second acquisition component 120 acquires point cloud data from the right side. Based on the point cloud data from the left and right sides of the body, a human body modeling algorithm is used to determine the initial point cloud model of the user's body.

[0100] For example, when a user wears the virtual fitting helmet 100, the extension directions of the first acquisition component 110 and the second acquisition component 120, which are in working condition, are parallel to the direction of the user's line of sight. In this case, for example, the first acquisition component 110 is located at the front of the user and the second acquisition component 120 is located at the back of the user. The first acquisition component 110 acquires point cloud data from the front of the user, and the second acquisition component 120 acquires point cloud data from the back of the user. Based on the point cloud data from the front and back of the user, a human body modeling algorithm is used to determine the initial point cloud model of the user's body.

[0101] It should be understood that the determined initial human body point cloud model can be used as the target human body point cloud model for displaying clothing models.

[0102] S150. Using the side repair algorithm, the initial human point cloud model is repaired to obtain the target human point cloud model.

[0103] Alternatively, the side repair algorithm is an algorithm based on Generative Adversarial Networks (GANs).

[0104] The lateral repair process can include: first, determining the density and curve features corresponding to the first point cloud data, and then determining the density and curve features corresponding to the second point cloud data. Next, feature encoding is performed on the density and curve features, and a Generative Adversarial Network (GAN) is used to learn the feature encoding and generate missing point cloud data for scanning blind spots. Finally, the target human point cloud model is determined based on the initial human point cloud model and the generated missing point cloud data.

[0105] The process of determining the density features can be as follows: For example, using a 3D grid size with an x-axis length of 0.03m, a y-axis length of 0.03m, and a z-axis length of 0.15m, the first point cloud data or the second point cloud data is divided. Assuming that N 3D grids are obtained, and each grid contains 0-M point data, then the number of points in each grid is taken as feature value 1, and the distance between all points in each grid is taken as feature value 2. Thus, an N×2 density feature matrix can be obtained.

[0106] The process of determining curve features may include: dividing the horizontal direction perpendicular to the line connecting the head and feet of the human body into P point cloud curves, each containing Q points. The first point of each point cloud curve is used to form an array S1 containing P points; the second point is used to form an array S2 containing P points, and so on, until the Qth point of each point cloud curve forms an array SQ containing P points. For each array, S data distribution characteristic values ​​for all points within the array are determined, thus obtaining a Q×S curve feature matrix.

[0107] The aforementioned N×2 density feature matrix and Q×S curve feature matrix are input into the generative adversarial network for training to obtain the missing point cloud data of the scanning blind zone.

[0108] It should be understood that the initial human point cloud model established using the first and second point cloud data is only a preliminary model, which may be incomplete and missing a lot of data, such as the point cloud data at the junction of the two. Therefore, in order to make the established model more complete, it is necessary to use the side repair algorithm to repair it.

[0109] Based on this, to facilitate calculation and achieve more accurate repair, the point cloud data in the initial human body point cloud model can be divided into four groups, for example, the point cloud data of the left half of the front side of the human body, the point cloud data of the right half of the front side of the human body, the point cloud data of the left half of the rear side of the human body, and the point cloud data of the right half of the rear side of the human body. Then, the point cloud data of the left half of the front side of the human body and the point cloud data of the left half of the rear side of the human body are used to repair the left side of the human body, and the point cloud data of the right half of the front side of the human body and the point cloud data of the left half of the rear side of the human body are used to repair the right side of the human body. Of course, the division can be further refined for more detailed repair; the specific division and calculation can be performed as needed, and this application embodiment does not impose any limitations on this.

[0110] It should also be understood that when a user is wearing a helmet, a target human body point cloud model can be built based on the first and second point cloud data acquired, and the target human body point cloud model can be continuously updated based on the point cloud data collected subsequently.

[0111] Based on the above, it should be understood that when the first initial data collected by the first acquisition component 110 and the second initial data collected by the second acquisition component 120 do not include the user's head information, the target human point cloud model established according to the above method will also not include the user's head. However, for better subsequent display effects, the user's head model can also be added to the target human point cloud model.

[0112] The following provides several exemplary descriptions of the process of generating the head model.

[0113] For example, the viewing angle of the data collected by the first acquisition component 110 and the second acquisition component 120 can be expanded so that the collected first initial data and second initial data include the user's head information. Thus, when establishing the target human body point cloud model, the point cloud model corresponding to the user's head can be included.

[0114] Example 2: Before wearing the virtual fitting helmet 100, the user takes a head photo. For example, the user raises the virtual fitting helmet 100 above their head and uses the first acquisition component 110 and the second acquisition component 120 to capture head information. The helmet body 130 then builds a head point cloud model based on the head information. Then, the head point cloud model is combined with the subsequently built human body point cloud model.

[0115] For example, a third method can be used to collect the user's head information by installing sensors on the helmet. The helmet body 130 then builds a head point cloud model based on the head information, and then combines the head point cloud model with the subsequently built human body point cloud model.

[0116] It should be understood that the above are only three examples, and the specific design and modification can be carried out as needed. This application does not impose any restrictions on these examples.

[0117] S160. Using a matching algorithm, determine one or more clothing models that match the target human body point cloud model.

[0118] It should be understood that the matching algorithm refers to comparing the size of the clothing model with the size of the target human body point cloud model one by one. This size may include at least one of the following: height, weight, shoulder width, chest circumference, waist circumference, sleeve length, and trouser length.

[0119] It should be understood that the clothing model library can be stored in the storage unit of the virtual fitting helmet 100, or it can be stored in other devices that communicate with the virtual fitting helmet 100. The clothing model library includes multiple clothing models and the corresponding size of each clothing model.

[0120] It should be understood that when a matching algorithm is not used to determine the clothing model that matches the target human point cloud model, if the user directly selects from a large number of clothing models, the size of the selected clothing model may not be suitable for the user. As a result, the selected clothing model cannot be well integrated with the target human point cloud model and will not be well displayed.

[0121] S170, In response to the user's second operation, combine the target clothing model onto the target human body point cloud model. The target clothing model is one or more clothing models selected by the user for display.

[0122] It should be understood that the user's second operation can be used to instruct the user to perform air gesture operations, voice operations, etc. Of course, the second operation can also be other operations, and can be set as needed. This application embodiment does not impose any restrictions on this. For example, the helmet body 130 of the virtual fitting helmet 100 stores a gesture recognition algorithm. When the user performs an air gesture operation, the first acquisition component 110 and the second acquisition component 120 recognize the user's air gesture from the acquired point cloud data and image data. Then, in response to the user's air gesture, human-computer interaction is performed, and the display content in the display device 200 is changed.

[0123] For example, in response to a user's index finger press, the selected clothing model is displayed on top of the target human point cloud model. In response to a user's index finger swipe, the displayed clothing model on the target human point cloud model is switched; in response to a user's open palm swipe, the display page is turned. In response to a user's two-finger swipe, the viewing angle is rotated; for example, after rotating the viewing angle, the user can see how the clothing model appears on the side or back of the target human point cloud model.

[0124] Here, the gestures of the air gesture can be set and changed as needed, and the corresponding commands of the air gesture can also be set and changed as needed. This application embodiment does not impose any restrictions on this.

[0125] S180. Based on the first initial data and the second initial data, use a human posture recognition algorithm to determine the target human posture category corresponding to the user, as well as the coordinate values ​​of key parts of the human body.

[0126] It should be understood that the human posture category may include at least one of standing, squatting, and sitting postures; the target human posture category is one of the aforementioned human posture categories.

[0127] It should be understood that key parts of the human body can be set as needed, such as the head, elbows, forearms, shoulders, hips, knees, and ankles. Specifically, they can be added and modified as needed, and this application embodiment does not impose any restrictions on this.

[0128] It should be understood that, based on the first initial data including the first point cloud data and the first image data, and the second initial data including the second point cloud data and the second image data, the human posture recognition algorithm is used to: perform preliminary identification of human posture categories based on the first image data and the second image data; and perform identification of human posture trajectories based on the first point cloud data and the second point cloud data.

[0129] Then, the features corresponding to the initially identified human posture categories and the features corresponding to the human posture trajectories are fused to obtain fused features located in three-dimensional spatial coordinates. The fused features are learned by an algorithm and compared with posture categories in a preset feature library to determine the target human posture category and the coordinate values ​​of key human body parts in three-dimensional spatial coordinates.

[0130] Based on this, the target clothing model can then be adjusted according to the target human body posture category and the coordinate values ​​of key human body parts.

[0131] It should also be understood that the initial recognition of human pose categories based on the first and second image data is performed on image sequence data. Each pixel in each frame of image data includes the values ​​of the three primary colors, thus each frame of image data includes data in three dimensions of the three primary colors RGB. The three primary colors are red, green, and blue.

[0132] Human posture trajectory recognition is performed based on point cloud sequence data, using the first and second point cloud data. Each point in each frame of point cloud data includes three coordinate values: x, y, and z. Therefore, each frame of point cloud data includes data in three dimensions: x, y, and z.

[0133] Based on this, when fusing the features corresponding to the initially identified human posture categories and the features corresponding to the human posture trajectories, the image sequence data and the point cloud sequence data can be fused frame by frame to obtain data in six dimensions: RGB and XYZ. In other words, the obtained fused features are data in six dimensions.

[0134] S190. Adjust the target clothing model according to the target human body posture category and the coordinate values ​​of key human body parts.

[0135] For example, if the target human posture is a squatting posture, and the coordinate values ​​of the key human body parts are j1 to j9, then based on the squatting posture and j1 to j9, the coordinate values ​​of the target clothing model are adjusted to match the current human posture, thus representing the wrinkles of the clothing when the person is squatting. Of course, depending on the different human postures, it can also represent the swaying state when the user is moving, or the state when unbuttoning clothes, etc.

[0136] S200: After the target clothing model is combined with the target human body point cloud model, it is displayed on the display device 200; after the target clothing model is adjusted, it is also displayed on the display device 200.

[0137] Users can try on clothes through the display device 200, experience the effect of the clothes on their body, and also change their body posture to see the corresponding display effect of the clothes in different postures. Based on the display effect, users can easily and conveniently determine their favorite clothes without physically trying them on.

[0138] The virtual try-on method provided in this application uses two sets of acquisition components on the helmet to collect human body data facing the user's viewpoint, thereby establishing a relatively complete human body point cloud model. Then, a matching algorithm is used to determine the clothing model that matches the human body point cloud model, allowing the user to select and display the desired clothing. The operation is simple. Because the collected data has a comprehensive coverage, the determined human body point cloud model is more accurate, resulting in better performance when matching clothing models.

[0139] This application also provides a computer-readable storage medium storing a computer program or instructions, which, when read and executed by a computer, causes the computer to perform the virtual try-on method.

[0140] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to execute the virtual try-on method described above.

[0141] The beneficial effects of the computer-readable storage medium and computer program product provided in this application embodiment are the same as the beneficial effects corresponding to the virtual try-on method described above, and will not be repeated here.

[0142] This application also provides an electronic device, such as... Figure 5 As shown, it includes: a processor; the processor executes a computer program stored in memory to implement the virtual try-on method as described in the embodiments of this application.

[0143] It should be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0144] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. Alternatively, the aforementioned processor can also be one or more integrated circuits used to control the execution of a program for the signal transmission method described above. The processing unit and the storage unit can be decoupled and respectively located on different physical devices, connected via wired or wireless means to realize their respective functions, thereby supporting the system chip in implementing the various functions described in the embodiments above. Alternatively, the processing unit and the memory can also be coupled to the same device.

[0145] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A virtual fitting system, characterized in that, include: Virtual fitting headsets and display devices; The virtual fitting helmet includes: a helmet body, and a first acquisition component and a second acquisition component installed on the helmet body; The first acquisition component and the second acquisition component unfold in opposite directions when the user wears the virtual fitting system, so that there is a certain distance between the first acquisition component and the second acquisition component and the user on the same horizontal plane. The first acquisition component is used to acquire the first initial data in the first viewpoint, and the second acquisition component is used to acquire the second initial data in the second viewpoint. The first viewpoint and the second viewpoint are opposite to each other and both face the user. The helmet body is used to determine the target human body point cloud model corresponding to the user based on the first initial data and the second initial data; The helmet body is also used to combine the target clothing model with the target human body point cloud model; The display device is used to display the combined target human body point cloud model and the target clothing model.

2. The virtual fitting system according to claim 1, characterized in that, The first acquisition component includes: a first lidar sensor and a first camera; the second acquisition component includes: a second lidar sensor and a second camera. The first lidar sensor is used to collect first point cloud data, the first camera is used to collect first image data, and the first initial data includes the first point cloud data and the first image data; The second lidar sensor is used to acquire second point cloud data, the second camera is used to acquire second image data, and the second initial data includes the second point cloud data and the second image data.

3. The virtual fitting system according to claim 2, characterized in that, The helmet body is also used to: determine an initial human point cloud model based on the first point cloud data and the second point cloud data using a human body modeling algorithm, and use the initial human point cloud model as the target human point cloud model. The human body modeling algorithm includes at least one of denoising, stitching, and registration.

4. The virtual fitting system according to claim 3, characterized in that, The helmet body is also used to: repair the initial human point cloud model using a side repair algorithm to obtain the target human point cloud model.

5. The virtual fitting system according to claim 3, characterized in that, The helmet body is also used to: correct the first three-dimensional spatial coordinates corresponding to the first lidar sensor based on the first point cloud data, and correct the coordinate values ​​of the first point cloud data using the corrected first three-dimensional spatial coordinates. The helmet body is also used to: correct the second three-dimensional spatial coordinates corresponding to the second lidar sensor based on the second point cloud data, and correct the coordinate values ​​of the second point cloud data using the corrected second three-dimensional spatial coordinates.

6. The virtual fitting system according to claim 5, characterized in that, The helmet body is also used to: segment the first point cloud data using a point cloud segmentation algorithm to generate first human point cloud data and first background point cloud data; Using a ground-based search algorithm, the first ground point cloud data is determined from the first background point cloud data; Based on the first ground point cloud data, the first three-dimensional spatial coordinates corresponding to the first lidar sensor are corrected. The helmet body is also used to: use the point cloud segmentation algorithm to segment the second point cloud data to generate second human body point cloud data and second background point cloud data; Using the ground-finding algorithm, the second ground point cloud data is determined from the second background point cloud data; Based on the second ground point cloud data, the second three-dimensional spatial coordinates corresponding to the second lidar sensor are corrected.

7. The virtual fitting system according to any one of claims 1 to 6, characterized in that, The helmet body is also used to: use a matching algorithm to determine one or more clothing models that match the target human body point cloud model.

8. The virtual fitting system according to any one of claims 1 to 6, characterized in that, The helmet body is also used to: determine the target human posture category and coordinate values ​​of key human body parts corresponding to the user based on the first initial data and the second initial data using a human posture recognition algorithm, wherein the human posture category includes at least one of standing posture, squatting posture, and sitting posture, and the target human posture category is one of the human posture categories; The shape of the target clothing model is adjusted according to the target human body posture category and the coordinate values ​​of the key parts of the human body; The display device is also used to display the adjusted target clothing model.

9. A virtual try-on method, characterized in that, The method, applied to the virtual fitting system according to claims 1 to 8, comprises: When a user wears the virtual fitting system, the first acquisition component acquires first initial data from a first perspective, and the second acquisition component acquires second initial data from a second perspective. The first perspective and the second perspective are opposite to each other and both face the user. Based on the first initial data and the second initial data, determine the target human body point cloud model corresponding to the user; The target clothing model is combined with the target human body point cloud model and displayed on the display device. The target clothing model is one or more clothing models, and the clothing model selected by the user for display is displayed.

10. A computer-readable storage medium storing a computer program or instructions, characterized in that, When the computer reads and executes the computer program or instructions, it causes the computer to implement the virtual try-on method as described in claim 9.

Images