Display method, device and terminal equipment

Abstract

The application provides a display method, a device and a terminal equipment. The display method comprises: obtaining a first background picture and a first depth image and a second depth image of a user; inputting the first depth image into a key point detection model to obtain first coordinate information of a hand of the user; inputting the second depth image into the key point detection model to obtain second coordinate information of the hand of the user; determining a first moving speed of a first fluid particle according to the first coordinate information and the second coordinate information, the fluid particle comprising the first fluid particle, a position of the first fluid particle in the first background picture corresponding to a position indicated by the second coordinate information; and displaying a first picture according to the first background picture and the first moving speed. By using the display method, the moving position of the hand of the user can be efficiently detected, and the moving effect of the corresponding fluid particle can be displayed, thereby improving the immersion and participation of the user in the creation of the display content.

Description

Technical Field

[0001] This application relates to the field of display technology, and more specifically, to a display method, apparatus, and terminal device. Background Technology

[0002] With the continuous development of display technology, users' demands for display products are also constantly increasing. Users are not only focused on larger screen sizes, but also have higher expectations for the level of intelligence in display devices. Against this backdrop, interactive display devices are becoming increasingly popular. When using interactive display devices, users can participate in the creation of displayed content through interaction, thereby enhancing immersion and engagement, transforming the viewing experience from "passive reception" to "active creation."

[0003] However, during user interaction with the display device, users typically need to wear specific devices (e.g., hand positioning devices) so that the display device can recognize the user's body position. Although this method can obtain user movement information relatively accurately, in actual use, wearing the device may cause many inconveniences, such as the device being bulky, cumbersome to wear, restricting freedom of movement, and even causing fatigue or discomfort after prolonged use, thus affecting the overall interactive experience and user engagement. Summary of the Invention

[0004] This application provides a display method, apparatus, and terminal device. By applying the above display method, the user's hand movement position can be detected efficiently, and the corresponding fluid particle movement effect can be displayed, thereby improving the user's immersion and participation in the creation of display content.

[0005] In a first aspect, a display method is provided, comprising: acquiring a first background image and a first depth image and a second depth image of a user, wherein the first background image includes fluid particles, and the acquisition time of the first depth image is earlier than the acquisition time of the second depth image; inputting the first depth image into a key point detection model to obtain first coordinate information of the user's hand; inputting the second depth image into the key point detection model to obtain second coordinate information of the user's hand; determining a first moving speed of a first fluid particle based on the first coordinate information and the second coordinate information, wherein the fluid particle includes the first fluid particle, and the position of the first fluid particle in the first background image corresponds to the position indicated by the second coordinate information; and displaying a first screen based on the first background image and the first moving speed, wherein the first screen includes a screen showing the movement of the first fluid particle.

[0006] In this embodiment, the movement speed of the first fluid particle is determined by the difference in coordinate information of the user's hand in two depth images, and the image of the first fluid particle moving with the user's hand is displayed. This method efficiently detects the user's hand movement and displays the corresponding fluid particle movement effect, thereby improving the user's immersion and participation in creating displayed content. Furthermore, by applying the above display method, the user does not need to wear specific equipment during interaction with the display device, thus improving the user's interactive experience.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, before acquiring the first background image and the user's first depth image and second depth image, the method includes: acquiring the user's third depth image and fourth depth image, wherein the acquisition time of the third depth image is earlier than the acquisition time of the fourth depth image; setting the pixel value of a first pixel in the third depth image to 0 to obtain the first depth image, wherein the first pixel is a pixel with a pixel value greater than a first preset distance; setting the pixel value of a second pixel in the fourth depth image to 0 to obtain the second depth image, wherein the second pixel is a pixel with a pixel value greater than the first preset distance.

[0008] In this embodiment of the application, by setting the pixel value of the first pixel in the third depth image and the fourth depth image to 0, the interference of non-interactive participants (e.g., people or objects) that are far away in the depth image can be reduced, the accuracy of the user's hand positioning can be improved, and thus the user's interactive experience can be further improved.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the pixel value of the third pixel in the fourth depth image is set to 0 to obtain a fifth image, wherein the third pixel is a pixel with a pixel value greater than a second preset distance, and the fifth image is a binary image indicating the contour region of the user; the fifth image is input into a particle system for processing to obtain a sixth image, wherein the contour region of the user is filled with particles in the sixth image; the step of displaying the first screen according to the first background image and the first moving speed includes: determining a seventh image according to the first background image and the first moving speed; and superimposing the sixth image and the seventh image to display the first screen.

[0010] In this embodiment, a depth image can be converted into a binary image, and the binary image can be processed using a particle system to fill the user's contour area with particles. Then, a sixth image including the particle-filled image is superimposed with a seventh image to display the first screen. In this way, the user's position can be displayed in real time during the interaction between the user and the display device, thereby further improving the user's immersion and participation in the creation of display content.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: acquiring an eighth image of the user, the eighth image being an RGB format image; determining third coordinate information of the user's head based on the eighth image; determining the position and trajectory of an object following or surrounding the user's head based on the third coordinate information; and superimposing the sixth image and the seventh image to display the first screen, which includes: superimposing the sixth image, the seventh image, and the position and trajectory of the object to display the first screen.

[0012] In this embodiment, an interactive effect can be generated based on the coordinate information of the user's head, making it appear as if an object (e.g., a butterfly or a bird) follows the user's head movement. This approach allows virtual content to be more synchronized with the user's physical actions, enhancing the user's immersion and sense of control over the displayed screen, thereby further improving the user's interactive experience.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the step of superimposing the sixth image, the seventh image, and the position and trajectory of the object to display the first screen includes: superimposing the sixth image, the seventh image, and the position and trajectory of the object to determine a second screen; inputting the second screen into a first neural network model to obtain the first screen, wherein the resolution of the first screen is higher than the resolution of the second screen; and displaying the first screen.

[0014] In this embodiment, since directly generating and displaying a high-resolution first image by overlaying the sixth image, the seventh image, and the object position and trajectory requires a lot of computing resources, a lower-resolution second image is generated first, and then the second image is processed by the first neural network model to obtain the first image. This can save the computing resources of the display device and support the display device to display a high-resolution first image, thereby further improving the user's interactive experience.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, before acquiring the first background image and the user's first depth image and second depth image, the method further includes: acquiring a second background image, the second background image including the fluid particles; and performing style transfer processing on the second background image to obtain the first background image.

[0016] In this embodiment of the application, by applying the style-transfer processed first background image to the generation of the first image, the artistic expression and visual appeal of the first image can be effectively enhanced. The generated first image retains the structural information of the original scene and incorporates the texture and color of a specific artistic style, thereby improving the overall visual experience of the image display.

[0017] Secondly, a display device is provided, comprising: an acquisition unit and a processing unit; the acquisition unit is configured to acquire a first background image and a first depth image and a second depth image of a user, the first background image including fluid particles, and the acquisition time of the first depth image being earlier than the acquisition time of the second depth image; the processing unit is configured to: input the first depth image into a key point detection model to obtain first coordinate information of the user's hand; input the second depth image into the key point detection model to obtain second coordinate information of the user's hand; determine a first moving speed of a first fluid particle based on the first coordinate information and the second coordinate information, the fluid particle including the first fluid particle, the position of the first fluid particle in the first background image corresponding to the position indicated by the second coordinate information; and display a first screen based on the first background image and the first moving speed, the first screen including a screen showing the movement of the first fluid particle.

[0018] For a description of the beneficial effects of the second aspect, please refer to the description of the beneficial effects of the first aspect, which will not be repeated here.

[0019] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to acquire the user's third depth image and fourth depth image, wherein the acquisition time of the third depth image is earlier than the acquisition time of the fourth depth image; the processing unit is further configured to: set the pixel value of the first pixel in the third depth image to 0 to obtain the first depth image, wherein the first pixel is a pixel whose pixel value is greater than a first preset distance; and set the pixel value of the second pixel in the fourth depth image to 0 to obtain the second depth image, wherein the second pixel is a pixel whose pixel value is greater than the first preset distance.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is further configured to: set the pixel value of the third pixel in the fourth depth image to 0 to obtain a fifth image, wherein the third pixel is a pixel with a pixel value greater than a second preset distance, and the fifth image is a binary image indicating the contour region of the user; input the fifth image into a particle system for processing to obtain a sixth image, wherein the contour region of the user is filled with particles in the sixth image; the processing unit is specifically configured to: determine a seventh image based on the first background image and the first moving speed; and superimpose the sixth image and the seventh image to display the first screen.

[0021] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to acquire an eighth image of the user, the eighth image being an RGB format image; the processing unit is further configured to: determine the third coordinate information of the user's head based on the eighth image; determine the position and trajectory of an object following or surrounding the user's head based on the third coordinate information; the processing unit is specifically configured to overlay the sixth image, the seventh image, and the position and trajectory of the object to display the first screen.

[0022] In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to: determine a second frame based on the sixth image, the seventh image, and the position and trajectory of the object; input the second frame into a first neural network model to obtain the first frame, wherein the resolution of the first frame is higher than the resolution of the second frame; and display the first frame.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to acquire a second background image, the second background image including the fluid particles; the processing unit is further configured to perform style transfer processing on the second background image to obtain the first background image.

[0024] Thirdly, a control device is provided, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory for reading and executing instructions in the memory, such that the device implements the method in any of the implementations of the first aspect described above.

[0025] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the method in any of the implementations of the first aspect described above.

[0026] Fifthly, a chip is provided, the chip including circuitry for performing the method in any of the implementations of the first aspect described above.

[0027] Sixthly, a computer program product is provided, the computer product including a computer program that, when the computer program is run by a processor, causes the method in any of the implementations of the first aspect to be executed.

[0028] In a seventh aspect, a terminal device is provided, comprising: a display device according to any one of the implementations of the second or third aspect described above. Attached Figure Description

[0029] Figure 1This is a schematic flowchart illustrating a display method provided in an embodiment of this application;

[0030] Figure 2 This is a schematic diagram of human body key point detection provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram illustrating the movement of fluid particles during user interaction with a display panel, provided in an embodiment of this application.

[0032] Figure 4 This is a schematic diagram of the outline display during the interaction between a user and a display panel, provided in an embodiment of this application;

[0033] Figure 5 This is a schematic diagram illustrating style transfer processing of a background image according to an embodiment of this application;

[0034] Figure 6 This is a schematic flowchart illustrating another display method provided in an embodiment of this application;

[0035] Figure 7 This is a schematic diagram of a process for particle filling of human body contours provided in an embodiment of this application;

[0036] Figure 8 This is a schematic diagram of a style transfer process provided in an embodiment of this application;

[0037] Figure 9 This is a schematic flowchart illustrating the effect of adding an object around a user's head, provided in an embodiment of this application.

[0038] Figure 10 This is a schematic diagram of an image fusion output provided in an embodiment of this application;

[0039] Figure 11 This is a schematic diagram of the structure of a first neural network model provided in an embodiment of this application;

[0040] Figure 12 This is a schematic diagram of training data augmentation during the training process of a first neural network model provided in an embodiment of this application;

[0041] Figure 13 This is a schematic diagram of a display device provided in an embodiment of this application;

[0042] Figure 14 This is a schematic diagram of another display device provided in an embodiment of this application. Detailed Implementation

[0043] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In this application, "at least one" means one or more, and "more" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0044] The use of prefixes such as "first" and "second" in this application embodiment is solely for distinguishing different descriptive objects and does not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is found in the claims or the context of the embodiments, and the use of such prefixes should not constitute unnecessary restrictions.

[0045] The embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to illustrate the principles of this application by way of example, but should not be used to limit the scope of this application, that is, this application is not limited to the described embodiments.

[0046] As described in the background section, currently, during user interaction with display devices, users typically need to wear specific devices (e.g., hand positioning devices) so that the display device can identify the user's limb position. Although this method can obtain user motion information relatively accurately, in actual use, wearing the device may cause many inconveniences, such as the device being bulky, cumbersome to wear, restricting freedom of movement, and even causing fatigue or discomfort after prolonged use, thus affecting the overall interactive experience and user engagement.

[0047] This application provides a display method, apparatus, and terminal device that can efficiently detect the user's hand movement position and display the corresponding fluid particle movement effect, thereby improving the user's immersion and participation in the creation of display content.

[0048] Figure 1 This is a schematic flowchart of a display method provided in an embodiment of this application. The execution subject of method 100 can be a terminal device, and method 100 can include steps S101 to S105.

[0049] S101, Obtain the first background image and the user's first depth image and second depth image.

[0050] The first background image includes fluid particles, and the first depth image was acquired earlier than the second depth image.

[0051] Optionally, the first depth image and the second depth image can be acquired by a motion-sensing camera or a depth camera, which may include a structured light, binocular stereo vision, or time-of-flight (ToF) motion-sensing camera.

[0052] S102, input the first depth image into the key point detection model to obtain the first coordinate information of the user's hand.

[0053] Optionally, the keypoint detection model can be the Kinect SDK skeleton detection model, for example, such as... Figure 2 As shown, after inputting the depth image into the model, the position coordinates of multiple human body key points can be obtained. From the position coordinates of multiple human body key points, the first coordinate information of the user's hand can be filtered out.

[0054] Optionally, the first coordinate information may include the coordinates of the user's left hand and / or right hand. For example, the first coordinate information may include: [Human Number 1][Left Hand][Coordinate 1], [Human Number 1][Right Hand][Coordinate 2]. In the case of multiple users, different human numbers can be used to distinguish them.

[0055] S103, input the second depth image into the key point detection model to obtain the second coordinate information of the user's hand.

[0056] Optionally, the second coordinate information may include the coordinates of the user's left hand and / or right hand. For example, the second coordinate information may include: [Human Number 1][Left Hand][Coordinate 3], [Human Number 1][Right Hand][Coordinate 4].

[0057] S104, determine the first moving velocity of the first fluid particle based on the first coordinate information and the second coordinate information.

[0058] The fluid particles include a first fluid particle, whose position in the first background image corresponds to the position indicated by the second coordinate information.

[0059] For example, the first moving speed can be determined by the following formula:

[0060] First movement speed = (Coordinates indicated by second coordinate information - Coordinates indicated by first coordinate information) × vscale

[0061] Here, vscale is the speed amplification factor, which can be set based on different display effects. For example, vscale = 15.

[0062] S105, based on the first background image and the first movement speed, display the first screen.

[0063] After obtaining the first moving speed, a speed attribute can be added to the first fluid particle in the first background image to obtain a first frame. This first frame includes the movement of the first particle. For example, the first frame can refer to... Figure 3 When adding speed attributes, the speed radius range can be set appropriately based on the display effect; for example, setting the speed radius range to 40.

[0064] Optionally, after adding velocity attributes to the first fluid particles, other attributes (e.g., density, gravity, pressure, or viscosity) can be added to obtain a fluid image. This fluid image is then blended with a first background image to obtain the first image. The blending of the fluid image and the first background image can be performed using the following formula:

[0065] First frame = First background image × alpha + Fluid image × (1-alpha)

[0066] Here, alpha is a blending parameter (transparency), and different values ​​can be set according to different effects, for example, alpha = 0.01.

[0067] Optionally, in step S105, the process of generating the fluid image and / or the first image can be implemented by a fluid simulation engine (e.g., WebGL, Processing, PixelFlow). This fluid simulation engine can utilize computer graphics technology to realistically simulate the visual effects of fluid movement such as liquids, smoke, flames, and oceans. Further optionally, the processing speed of the fluid simulation engine can be greater than or equal to 20fps, and the vertical resolution of the first image can be greater than or equal to 540.

[0068] In this embodiment, the movement speed of the first fluid particle is determined by the difference in coordinate information of the user's hand in two depth images, and the image of the first fluid particle moving with the user's hand is displayed. This method efficiently detects the user's hand movement and displays the corresponding fluid particle movement effect, thereby improving the user's immersion and participation in creating displayed content. Furthermore, by applying the above display method, the user does not need to wear specific equipment during interaction with the display device, thus improving the user's interactive experience.

[0069] According to some embodiments, before step S101, method 100 further includes: acquiring a third depth image and a fourth depth image of the user, wherein the acquisition time of the third depth image is earlier than the acquisition time of the fourth depth image; setting the pixel value of a first pixel in the third depth image to 0 to obtain a first depth image, wherein the first pixel is a pixel whose pixel value is greater than a first preset distance; setting the pixel value of a second pixel in the fourth depth image to 0 to obtain a second depth image, wherein the second pixel is a pixel whose pixel value is greater than a first preset distance.

[0070] In this embodiment of the application, by setting the pixel value of the first pixel in the third depth image and the fourth depth image to 0, the interference of non-interactive participants (e.g., people or objects) that are far away in the depth image can be reduced, the accuracy of the user's hand positioning can be improved, and thus the user's interactive experience can be further improved.

[0071] Optionally, the third and fourth depth images can be acquired by a motion-sensing camera or a depth camera. The first and second depth images obtained through the above processing method can also be referred to as foreground depth maps.

[0072] According to some embodiments, method 100 further includes: setting the pixel value of the third pixel in the fourth depth image to 0 to obtain a fifth image, wherein the third pixel is a pixel whose pixel value is greater than a second preset distance, and the fifth image is a binary image indicating the contour region of the user; inputting the fifth image into a particle system for processing to obtain a sixth image, wherein the contour region of the user is filled with particles in the sixth image; step S105 includes: determining a seventh image based on a first background image and a first moving speed; superimposing the sixth image and the seventh image to display a first screen.

[0073] In this embodiment, a depth image can be converted into a binary image, and the binary image can be processed using a particle system to fill the user's contour area with particles. Then, a sixth image including the particle-filled image is superimposed on a seventh image to display the first screen. In this way, the user's position can be displayed in real time during interaction with the display device, thereby further enhancing the user's immersion and participation in creating displayed content.

[0074] For example, the sixth image can be referenced Figure 4 The seventh image can be the fluid image generated above.

[0075] Optionally, the values ​​of the first preset distance and the second preset distance can be the same or different, depending on the display requirements and application scenario.

[0076] Optionally, during the binarization process, pixel values ​​in the fourth depth image that are smaller than the second preset distance can be set to the maximum value (1 or 255) to separate the background and foreground, thereby obtaining the fifth image.

[0077] It should be understood that particle systems are a technique in computer graphics used to simulate the irregular motion of irregular bodies composed of a large number of tiny particles in nature, and to render realistic graphics.

[0078] According to some embodiments, method 100 further includes: acquiring an eighth image of the user, the eighth image being an image in RGB format; determining third coordinate information of the user's head based on the eighth image; determining the position and trajectory of an object following or surrounding the user's head based on the third coordinate information; step S105 includes: superimposing the sixth image, the seventh image, and the position and trajectory of the object to display a first screen.

[0079] In this embodiment, an interactive effect can be generated based on the coordinate information of the user's head, making it appear as if an object (e.g., a butterfly or a bird) follows the user's head movement. This approach allows virtual content to be more synchronized with the user's physical actions, enhancing the user's immersion and sense of control over the displayed screen, thereby further improving the user's interactive experience.

[0080] Optionally, the eighth image can be acquired by an RGB camera, and the RGB camera, depth camera, and motion sensor camera can be integrated into the same acquisition device.

[0081] Optionally, when generating multiple objects moving in the same manner, the simulation of cooperative behavior can be achieved based on a swarm motion simulation model (such as the Boids algorithm). This swarm motion simulation model simulates the collective movement characteristics of flocks of birds, schools of fish, etc., in nature, causing individuals to follow certain behavioral rules during movement, thus presenting a coordinated dynamic effect. The Boids algorithm can include three constraints: separation, used to avoid individuals getting too close to prevent collisions; alignment, used to make the movement direction of individuals tend to be consistent with the surrounding individuals; and distance, used to keep individuals relatively close within the group, maintaining the overall compactness of the group. The combination of these three constraints jointly determines the motion state of the object. In addition, to achieve the effect of a biological swarm following the user's movement, a following constraint can be added, setting the center coordinates of the user's head as the following target.

[0082] According to some embodiments, superimposing a sixth image, a seventh image, and the position and trajectory of an object to display a first screen includes: determining a second screen based on the superposition of the sixth image, the seventh image, and the position and trajectory of the object; inputting the second screen into a first neural network model to obtain the first screen, wherein the resolution of the first screen is higher than the resolution of the second screen; and displaying the first screen.

[0083] In this embodiment, since directly generating and displaying a high-resolution first image by overlaying the sixth image, the seventh image, and the object position and trajectory requires a lot of computing resources, a lower-resolution second image is generated first, and then the second image is processed by the first neural network model to obtain the first image. This can save the computing resources of the display device and support the display device to display a high-resolution first image, thereby further improving the user's interactive experience.

[0084] According to some embodiments, before step S101, method 100 further includes: obtaining a second background image, the second background image including the fluid particles; performing style transfer processing on the second background image to obtain the first background image.

[0085] In this embodiment of the application, by applying the style-transfer processed first background image to the generation of the first image, the artistic expression and visual appeal of the first image can be effectively enhanced. The generated first image retains the structural information of the original scene and incorporates the texture and color of a specific artistic style, thereby improving the overall visual experience of the image display.

[0086] For example, such as Figure 5 As shown, Figure 5 (a) in the image is the second background image. Figure 5 (b) in the image is the first background image obtained after style transfer. Figure 5 It can be seen that the first background image after style transfer can increase artistic expression and visual appeal.

[0087] It should be understood that, unless otherwise specified or logically conflicting, the terminology and / or descriptions in the various embodiments of this application are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. For example, Figure 6 This is a schematic flowchart illustrating another display method provided in an embodiment of this application. Figure 6 In this process, the terminal device can perform image fusion based on the seventh image output by the fluid simulation engine, the sixth image output by the particle system, the first background image obtained through style transfer, and the position and trajectory of the object following the movement, generating a second image with a lower resolution. This second image is then subjected to super-resolution processing to obtain the first image, which is then displayed on a large screen. Furthermore, in... Figure 6In this process, images required for hand motion detection are acquired through a motion-sensing camera, images required for human contour detection are acquired through a depth camera, and images required for style transfer and head detection are acquired through an RGB camera. This approach enables targeted and efficient data acquisition, thereby improving the overall perception accuracy and performance of the display device.

[0088] The following is combined Figure 7 This document details the process of using particle filling to fill the human body contour in Method 100 to generate the sixth image. Figure 7 This is a schematic diagram of a process for particle filling of human body contours provided in an embodiment of this application.

[0089] like Figure 7 As shown, the fourth depth image can be binarized to obtain the fifth image, and then blob detection can be performed on the fifth image. Blob detection refers to identifying connected regions in an image that possess unique features, differing significantly from other regions in brightness and color, texture, area, shape, and other characteristics. During blob detection, desired blobs can be screened out by setting certain constraints, including brightness, area, and shape. Shape constraints are typically represented by roundness, eccentricity, and convexity.

[0090] When a pixel's position lies within the outline of a blob, the area is filled with particles using a particle system. Based on the desired effect and the hardware capabilities of the terminal device, the number, size range, speed range, and color of the particles are set to generate the sixth image. When filling the blob with particles, if the new position of a single particle is not within the blob's outline, a position within the blob's outline can be randomly initialized as the particle's new position.

[0091] Optionally, the limitations for speckle detection may include setting the speckle area to be greater than one-twelfth of the area of ​​the fourth depth image, and limiting the number of speckle edges to greater than 100.

[0092] Alternatively, blob detection can be performed using the SimpleBlobDetector algorithm from the OpenCV library or the BlobDetection library from the Processing language environment.

[0093] Optionally, the vertical resolution of the sixth image generated after particle filling can be set between 720 and 1080 pixels per inch (PPI).

[0094] The following is combined Figure 8 A detailed explanation of the style transfer process in Method 100. Figure 8 This is a schematic diagram of a style transfer process provided in an embodiment of this application.

[0095] like Figure 8 As shown, the second background image can be input into the content encoder to generate content features, the style image can be input into the style encoder to generate style features, the content features and style features can be processed by the feature matching module, and the processing result can be output to the decoder to obtain the first background image.

[0096] Optionally, the content encoder and style encoder may include a pre-trained VGG-19 encoder. To improve the processing speed of style transfer, VGG-19 can undergo joint compression processing involving model pruning, distillation, and quantization to reduce computational load and improve running speed. Pruning and distillation can effectively reduce GPU memory consumption, but their speed improvement is limited for high-resolution second background images. Therefore, to achieve real-time processing, a lightweight network structure can also be selected as the encoder, such as a miniature / lightweight encoder based on MobileNet.

[0097] Optionally, when the style images are predetermined, one or more fixed style images can be pre-generated, and style features can be output via a style encoder and stored. When style transfer is required on the second background image, the pre-stored style features can be used directly, thereby eliminating the need for calculation by the style encoder and improving the processing speed of style transfer.

[0098] Optionally, the content encoder and style encoder can be of the same or different types. For example, VGG-19 can be used as the style encoder, while a MobileNet-based mini-encoder can be used as the content encoder. The reason for this is that VGG-19 performs well as a style encoder in extracting image style features, but it has a large computational load, memory usage, and runtime. Therefore, when used as a style encoder, the calculation results of the style features can be stored in advance to omit this operation.

[0099] Optionally, the feature matching module can also be called the feature modulation module. The feature matching module can use methods such as adaptive instance normalization (AdaIN), dynamic instance normalization (DIN), or adaptive attention normalization (AdaAttN) to match content features and style features.

[0100] Optionally, the decoder can adopt a structure corresponding to the content encoder to gradually restore the content features extracted during the encoding process, ensuring that the generated first background image retains its stylistic features while preserving the structural and semantic information of the second background image to the greatest extent.

[0101] Optionally, to improve the quality of style transfer, style loss and content loss functions can be introduced during model training. The style loss function measures style consistency by calculating the difference in the Gram matrix between the generated image and the target style image in the feature space; the content loss measures the difference in mid-level features between the generated image and the content image, ensuring the preservation of semantic structure. The weighted combination of the two helps to achieve effective fusion of style and content, thereby improving the visual quality and stability of the transfer results. In addition, other loss functions can be added during model training. For example, adding total variational loss can make the generated first background image more spatially smooth and continuous. Furthermore, adding cross-image similarity loss can make style transfer smoother, more continuous, and more stable in time, avoiding flickering and artifacts.

[0102] The following is combined Figure 9 This section details the process of adding an object effect around the user's head, as described in Method 100. Figure 9 This is a schematic flowchart of an embodiment of the present application providing an effect of adding an object around a user's head. Method 900 may include steps S901 to S904.

[0103] S901, obtain the third coordinate information of the user's head.

[0104] S902 sets the values ​​for the three constraints (separate, aligned, and clustered) of the Boids algorithm.

[0105] For example, values ​​can be set for the three constraints based on preset effects to obtain separation weight, alignment weight, aggregation weight, and first volume velocity, etc.

[0106] S903, calculate and add follow constraint.

[0107] For example, when the added object is a group of animals, for each individual in the group, the position difference vector between each individual and the target being followed, and the following traction vector can be calculated. After obtaining the following traction vector, the second individual velocity of each individual can be determined based on the following migration vector and the first body velocity of each individual.

[0108] Alternatively, the position difference vector can be calculated using the following formula:

[0109] The position difference vector is calculated as [(head coordinate x – individual position x), (head coordinate x – individual position y)]. After obtaining the position difference vector, it can be normalized.

[0110] Alternatively, the following traction vector can be calculated using the following formula:

[0111] Following traction vector = Normalized position difference vector – Current individual velocity

[0112] Alternatively, the second volume velocity can be obtained using the following formula:

[0113] Second individual velocity = Following traction vector × Following weight + First individual velocity

[0114] S904, updated graphics.

[0115] For example, the movement trajectory of an individual can be determined based on the position of each individual and the speed of the second individual, thereby displaying the first screen.

[0116] The following is combined Figure 10 This section details the process in method 100 of overlaying the sixth and seventh images with the object's position and trajectory to display the first screen. Figure 10 This is a schematic diagram of an image fusion output provided in an embodiment of this application.

[0117] like Figure 10 As shown, the image output by the style transfer module (the first background image) can be used as the first background. The seventh image based on the fluid simulation engine is applied to the first background to obtain the second background. The second background and the sixth image output by the particle system are overlaid to obtain the third background. The third background is overlaid with the position and trajectory of the object output by the physical tracking motion to complete the image fusion output (i.e., output the first or second image).

[0118] Optionally, when performing image fusion, for input images with different resolutions, interpolation upsampling can be used to scale them, unify the image resolution, and set the output resolution of the image fusion to the highest resolution of the input image.

[0119] The following is combined Figure 11 The structure and training process of the first neural network model in Method 100 are described in detail. Figure 11 This is a schematic diagram of the structure of a first neural network model provided in an embodiment of this application.

[0120] like Figure 11 As shown, the first neural network model may include an input image, multiple convolutional layers, a depth-to-space layer, an addition operation, and an output image.

[0121] The kernel size of the convolutional layer can be set to 3×3, the number of channels can be set to 4, and the number of convolutional layers can be greater than or equal to 3 (determined based on the GPU computing power of the terminal device). The main reason for setting the number of channels to 4 is that GPU shaders typically perform parallel computations in units of 4 channels when processing tensor data. By dividing the tensor into 4 channels, it not only better adapts to the hardware characteristics of the GPU but also effectively reduces computational overhead, latency, and power consumption, thereby meeting the real-time processing requirements during interaction and further improving overall performance and processing speed.

[0122] Depth-to-spatial layers can rearrange the channel dimensions of feature maps (the output of convolutional layers) into spatial dimensions, thereby achieving upsampling and resolution enhancement of feature maps.

[0123] The addition operation is used to fuse multiple feature maps, thereby enhancing the expressive power of the model by integrating feature information from different layers or paths.

[0124] The input image can refer to the second screen, and the output image can refer to the first screen. Optionally, the width and height of the first screen can be twice that of the second image.

[0125] Optionally, the loss function used by the first neural network during model training can be a weighted sum of L1 and learned perceptual image patch similarity (LPIPS). This setting of the loss function, compared to using L1 or L2 as the loss function alone, can make the output of the neural network model closer to human perception.

[0126] In one possible implementation, after the neural network model is trained using a deep learning framework (e.g., PyTorch, TensorFlow), the model parameters (e.g., weights and biases) can be exported. The network structure of the model can then be implemented using OpenGL GLSL, and the model parameters imported. The reason for using OpenGL GLSL to implement this network structure is that it uses GPU shaders for computation, is not limited by hardware, and supports GPUs from different manufacturers and mobile devices. Furthermore, OpenGL GLSL can fully utilize the characteristics of the network structure to meet processing speed and performance requirements. In addition, by using GPU shaders for computation, OpenGL GLSL can directly render the super-resolution image to the screen, fully utilizing hardware features and avoiding additional memory and video memory copying.

[0127] In one possible implementation, the first network model, during model training, can augment the training data by overlaying random 2D points, lines, or shapes onto the training images (e.g., overlaying...). Figure 12 (as shown by the random lines), thereby specifically improving the resolution of the model's output image.

[0128] Figure 13 This is a schematic diagram of a display device provided in an embodiment of this application. The device 1300 may include an acquisition unit 1310, a storage unit 1320, and a processing unit 1330. The acquisition unit 1310 is used to acquire instructions and / or data, the storage unit 1320 is used to implement corresponding storage functions and store corresponding instructions and / or data, and the processing unit 1330 is used to perform data processing so that the device 1300 can implement the aforementioned display method.

[0129] According to some embodiments, the display device 1300 includes: an acquisition unit 1310 and a processing unit 1330; the acquisition unit 1310 is used to acquire a first background image and a first depth image and a second depth image of the user, the first background image including fluid particles, and the acquisition time of the first depth image being earlier than the acquisition time of the second depth image; the processing unit 1330 is used to: input the first depth image into a key point detection model to obtain first coordinate information of the user's hand; input the second depth image into the key point detection model to obtain second coordinate information of the user's hand; determine a first moving speed of the first fluid particles based on the first coordinate information and the second coordinate information, the fluid particles including first fluid particles, the position of the first fluid particles in the first background image corresponding to the position indicated by the second coordinate information; and display a first screen based on the first background image and the first moving speed, the first screen including the screen of the first fluid particles moving.

[0130] In one possible implementation, the acquisition unit 1310 is further configured to acquire the user's third depth image and fourth depth image, wherein the acquisition time of the third depth image is earlier than the acquisition time of the fourth depth image; the processing unit 1330 is further configured to: set the pixel value of the first pixel in the third depth image to 0 to obtain a first depth image, wherein the first pixel is a pixel whose pixel value is greater than a first preset distance; and set the pixel value of the second pixel in the fourth depth image to 0 to obtain a second depth image, wherein the second pixel is a pixel whose pixel value is greater than a first preset distance.

[0131] In one possible implementation, the processing unit 1330 is further configured to: set the pixel value of the third pixel in the fourth depth image to 0 to obtain a fifth image, wherein the third pixel is a pixel whose pixel value is greater than a second preset distance, and the fifth image is a binary image indicating the user's contour region; input the fifth image into the particle system for processing to obtain a sixth image, wherein the user's contour region is filled with particles in the sixth image; the processing unit 1330 is specifically configured to: determine a seventh image based on the first background image and the first moving speed; and superimpose the sixth image and the seventh image to display the first screen.

[0132] In one possible implementation, the acquisition unit 1310 is further configured to acquire the user's eighth image, which is an RGB format image; the processing unit 1330 is further configured to: determine the third coordinate information of the user's head based on the eighth image; determine the position and trajectory of an object following or surrounding the user's head based on the third coordinate information; and the processing unit 1330 is specifically configured to overlay the sixth image, the seventh image, and the position and trajectory of the object to display the first screen.

[0133] In one possible implementation, the processing unit 1330 is specifically used to: determine a second frame based on the sixth image, the seventh image, and the position and trajectory of the object; input the second frame into a first neural network model to obtain a first frame, wherein the resolution of the first frame is higher than that of the second frame; and display the first frame.

[0134] In one possible implementation, the acquisition unit 1310 is further configured to acquire a second background image, the second background image including fluid particles; the processing unit 1330 is further configured to perform style transfer processing on the second background image to obtain a first background image.

[0135] Figure 14 This is a schematic diagram of another display device provided in an embodiment of this application.

[0136] The device 1400 includes a memory 1410, a processor 1420, and a communication interface 1430. The memory 1410, processor 1420, and communication interface 1430 are connected via an internal connection path. The memory 1410 stores instructions, and the processor 1420 executes the instructions stored in the memory 1410 to control the communication interface 1430 to acquire information, thereby enabling the device 1400 to implement the aforementioned display method. Optionally, the memory 1410 can be coupled to the processor 1420 via an interface, or it can be integrated with the processor 1420.

[0137] It should be noted that the communication interface 1430 described above uses a transceiver device, such as, but not limited to, a transceiver. The communication interface 1430 may also include an input / output interface.

[0138] The processor 1420 stores one or more computer programs, which include instructions. When the instructions are executed by the processor 1420, the display device 1400 performs the display methods described in the above embodiments.

[0139] In implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 1420 or by instructions in software form. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1410, and the processor 1420 reads the information in memory 1410 and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0140] This application also provides a computer-readable storage medium storing program code that, when executed on a computer, causes the computer to perform the above-described... Figure 1 , Figures 6 to 11 The method shown in any one of them.

[0141] This application also provides a computer program product, which includes a computer program that, when run, causes the computer to perform the above-described actions. Figure 1 , Figures 6 to 11 The method shown in any one of them.

[0142] This application embodiment also provides a chip, including: a circuit, the circuit being used to perform the above... Figure 1 , Figures 6 to 11 The method shown in any one of them.

[0143] This application embodiment also provides a terminal device, the terminal device including: as shown in the example. Figure 13 or Figure 14 The display device shown.

[0144] Optionally, terminal devices include, but are not limited to: monitors, all-in-one conference machines, televisions, projectors, interactive whiteboards, smart tablets, and other multimedia terminal devices with audio and video input and output capabilities.

[0145] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0148] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0150] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A display method, characterized in that, The method includes: Acquire a first background image and a first depth image and a second depth image of the user. The first background image includes fluid particles, and the acquisition time of the first depth image is earlier than the acquisition time of the second depth image. The first depth image is input into the key point detection model to obtain the first coordinate information of the user's hand; The second depth image is input into the key point detection model to obtain the second coordinate information of the user's hand; Based on the first coordinate information and the second coordinate information, a first moving speed of the first fluid particle is determined. The fluid particle includes the first fluid particle, and the position of the first fluid particle in the first background image corresponds to the position indicated by the second coordinate information. Based on the first background image and the first movement speed, a first screen is displayed, which includes a screen showing the movement of the first fluid particles.

2. The method as described in claim 1, characterized in that, Before acquiring the first background image and the user's first depth image and second depth image, the method includes: Acquire the user's third depth image and fourth depth image, wherein the acquisition time of the third depth image is earlier than the acquisition time of the fourth depth image; The pixel value of the first pixel in the third depth image is set to 0 to obtain the first depth image, where the first pixel is a pixel whose pixel value is greater than a first preset distance; The pixel value of the second pixel in the fourth depth image is set to 0 to obtain the second depth image, where the second pixel is a pixel whose pixel value is greater than the first preset distance.

3. The method as described in claim 2, characterized in that, The method further includes: The pixel value of the third pixel in the fourth depth image is set to 0 to obtain the fifth image. The third pixel is a pixel whose pixel value is greater than the second preset distance. The fifth image is a binary image that indicates the contour region of the user. The fifth image is input into the particle system for processing to obtain a sixth image, in which the user's contour region is filled with particles. The step of displaying the first screen based on the first background image and the first movement speed includes: The seventh image is determined based on the first background image and the first movement speed; The sixth and seventh images are superimposed to display the first screen.

4. The method as described in claim 3, characterized in that, The method further includes: Obtain the user's eighth image, which is an RGB format image; Based on the eighth image, determine the third coordinate information of the user's head; Based on the third coordinate information, determine the position and trajectory of the object following or surrounding the user's head; The step of overlaying the sixth image and the seventh image to display the first screen includes: The first screen is displayed by overlaying the sixth image, the seventh image, and the position and trajectory of the object.

5. The method as described in claim 4, characterized in that, The step of overlaying the sixth image, the seventh image, and the position and trajectory of the object to display the first screen includes: The second frame is determined based on the sixth image, the seventh image, and the position and trajectory of the object; The second image is input into the first neural network model to obtain the first image, wherein the resolution of the first image is higher than that of the second image; The first screen is displayed.

6. The method according to any one of claims 1 to 5, characterized in that, Before acquiring the first background image and the user's first depth image and second depth image, the method further includes: Obtain a second background image, the second background image including the fluid particles; The second background image is subjected to style transfer processing to obtain the first background image.

7. A display device, characterized in that, The display device includes: an acquisition unit and a processing unit; The acquisition unit is used to acquire a first background image and a first depth image and a second depth image of the user. The first background image includes fluid particles, and the acquisition time of the first depth image is earlier than the acquisition time of the second depth image. The processing unit is used for: The first depth image is input into the key point detection model to obtain the first coordinate information of the user's hand; The second depth image is input into the key point detection model to obtain the second coordinate information of the user's hand; Based on the first coordinate information and the second coordinate information, a first moving speed of the first fluid particle is determined. The fluid particle includes the first fluid particle, and the position of the first fluid particle in the first background image corresponds to the position indicated by the second coordinate information. Based on the first background image and the first movement speed, a first screen is displayed, which includes a screen showing the movement of the first fluid particles.

8. The apparatus as claimed in claim 7, characterized in that, The acquisition unit is also used to acquire the user's third depth image and fourth depth image, wherein the acquisition time of the third depth image is earlier than the acquisition time of the fourth depth image; The processing unit is further configured to: The pixel value of the first pixel in the third depth image is set to 0 to obtain the first depth image, where the first pixel is a pixel whose pixel value is greater than a first preset distance; The pixel value of the second pixel in the fourth depth image is set to 0 to obtain the second depth image, where the second pixel is a pixel whose pixel value is greater than the first preset distance.

9. The apparatus as claimed in claim 8, characterized in that, The processing unit is further configured to: The pixel value of the third pixel in the fourth depth image is set to 0 to obtain the fifth image. The third pixel is a pixel whose pixel value is greater than the second preset distance. The fifth image is a binary image that indicates the contour region of the user. The fifth image is input into the particle system for processing to obtain a sixth image, in which the user's contour region is filled with particles. The processing unit is specifically used for: The seventh image is determined based on the first background image and the first movement speed; The sixth and seventh images are superimposed to display the first screen.

10. The apparatus as claimed in claim 9, characterized in that, The acquisition unit is further configured to acquire the user's eighth image, wherein the eighth image is an image in RGB format; The processing unit is further configured to: Based on the eighth image, determine the third coordinate information of the user's head; Based on the third coordinate information, determine the position and trajectory of the object following or surrounding the user's head; The processing unit is specifically used to overlay the sixth image, the seventh image, and the position and trajectory of the object to display the first screen.

11. The apparatus as claimed in claim 10, characterized in that, The processing unit is specifically used for: The second frame is determined based on the sixth image, the seventh image, and the position and trajectory of the object; The second image is input into the first neural network model to obtain the first image, wherein the resolution of the first image is higher than that of the second image; The first screen is displayed.

12. The apparatus according to any one of claims 7 to 11, characterized in that, The acquisition unit is further configured to acquire a second background image, the second background image including the fluid particles; The processing unit is further configured to perform style transfer processing on the second background image to obtain the first background image.

13. A display device, characterized in that, The device includes a processor and a memory, the processor being coupled to the memory, the memory being used to store computer programs or instructions, and the processor being used to execute the computer programs or instructions in the memory, such that the method of any one of claims 1 to 6 is performed.

14. A terminal device, characterized in that, include: The display device as claimed in any one of claims 7 to 13.

Images