A display method, system, electronic device and storage medium based on character motion state

Abstract

The embodiment of the present invention discloses a display method, system, electronic device and storage medium based on the motion state of a person, wherein the method includes: obtaining a naked-eye 3D scene projected by a projection device; obtaining video data of a target person; extracting the posture features of the target person in the video data; calculating the motion speed and the first motion direction of the target person according to the posture features; determining the first floor tile according to the first motion direction, and controlling the first floor tile to move to an empty area; controlling the second floor tile in (N‑2)×(N‑2) sub-block areas to move in the first direction according to the first motion direction and the motion speed, and the first motion direction is opposite to the first direction; adjusting the picture projected by the projection device according to the first motion direction and the motion speed, so that the naked-eye 3D scene moves at the motion speed relative to the center point in the first area, and the moving direction is the first direction. Implementation of the embodiment of the present application can increase the range of interactive scenes and enhance the user's immersive experience.

Description

Technical Field

[0001] The present invention relates to the field of digital intelligence services, specifically to the field of digital intelligence services for three-dimensional models, and in particular to a display method, system, electronic device and storage medium based on the motion state of a character. Background Art

[0002] Although the current immersive cinema technology has brought a new immersive experience to users, there are still many challenges in the interaction between people and devices. For example, due to the current space limitations, the interaction between characters and interactive systems is limited to the set space; it is unable to provide users with an immersive interactive experience. Summary of the invention

[0003] The present invention provides a display method based on character motion status, a 9D interactive system, an electronic device and a storage medium, which can increase the scope of interactive scenes and enhance the immersive experience of users.

[0004] On the one hand, the present invention provides a display method based on the motion state of a character, the method comprising: the method is applied to a 9D interactive system, the 9D interactive system comprising a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environment control device, a blowing device, and a vibration feedback device;

[0005] The 9D interactive system is deployed in a first area, the first area is a square, the first area is composed of N×N sub-square areas, N is a positive odd number and N≥3, each sub-square area is the same size as a floor tile, the number of floor tiles included in the first area is less than N×N, and the 9D interactive system can control the floor tile to move to any of the sub-square areas in the first area;

[0006] In an initial state, the outermost circle of the N×N sub-block areas is provided with floor tiles; the outermost circle of the (N-1)×(N-1) sub-block areas centered on the central sub-block is an empty area; the (N-2)×(N-2) sub-block areas centered on the central sub-block are all provided with floor tiles; the target person is located in the central sub-block in the initial state; the method comprises:

[0007] The naked eye 3D scene projected by the projection device;

[0008] Acquire video data of the target person;

[0009] Extracting posture features of the target person in the video data;

[0010] Calculate the movement speed and the first movement direction of the target person according to the posture feature;

[0011] Determine a first floor tile according to the first movement direction, control the first floor tile to move to the empty area, the first floor tile being the floor tile of the outermost circle of the N×N sub-block area in the initial state;

[0012] The second floor tile in the (N-2)×(N-2) sub-block area is controlled to move in a first direction according to the first movement direction and the movement speed, and the first movement direction is opposite to the first direction; the picture projected by the projection device is adjusted according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point of the first area, and the movement direction is the first direction.

[0013] It should be noted that the 9D interactive system can determine the moving distance and moving time of the human body based on the time interval between adjacent video frames and the posture features between adjacent video frames, and then calculate the movement speed and the first movement direction.

[0014] Optionally, 9D refers to naked eye 3D, vibration, floor tiles moving forward and backward, left and right, rotation, smell, wind, rain and snow.

[0015] In a possible implementation, the extraction of the posture features of the target person in the video data; the calculation of the movement speed and the first movement direction of the target person according to the posture features, including: obtaining an input image according to the video data, inputting the image into an image segmentation model for image segmentation, and obtaining a target person mask; performing target posture detection through the target person mask to obtain the target person's step posture; determining the movement speed and the first movement direction of the target person according to the target person's step posture; wherein the image segmentation model is composed of three layers, the first layer includes a first feature prediction module and a first residual calculation module, the second layer includes a second feature prediction module and a second residual calculation module, and the third layer includes a third feature prediction module and a third residual calculation module; the first residual calculation module, the second residual module and the third residual module are connected in sequence, and the residual feature map of the layer is extracted; the third feature prediction module, the second feature prediction module and the first feature prediction module are connected in sequence, and the target prediction feature map of the layer is obtained, wherein the output of the first feature prediction module as a target person mask; the training method of the image segmentation model is: inputting the image into the first residual calculation module to obtain a first residual feature map, inputting the first residual feature map into the second residual calculation module to obtain a second residual feature map, and inputting the second residual feature map into the third residual calculation module to obtain a third residual feature map; calculating the sum of the first residual feature map, the second residual feature map and the third residual feature map, and calculating the self-supervised loss of the image segmentation model according to the sum of the above residual feature maps as the first loss function; inputting the third residual feature map into the third feature prediction module to obtain a third target prediction feature map; upsampling the third target prediction feature map and the second residual feature map into the second feature prediction module to obtain a second target prediction feature map; fusing the upsampled third target prediction feature map and the second target prediction feature map, and calculating the second loss function according to the difference between the upsampled third target prediction feature map and the real segmentation mask; adding the first loss function and the second loss function to obtain the target segmentation loss function; using the target segmentation loss function to train the initial segmentation module to obtain the image segmentation model.

[0016] It should be noted that the basic idea of ​​Dice loss is to calculate the overlap between the predicted results and the true results, and optimize the model by minimizing the difference between the two; Focal loss handles the class imbalance problem by assigning more weights to examples that are difficult to classify or easily misclassified and reducing the weights of simple examples; the second loss function can be calculated using the sum of Dice loss and Focal loss to achieve a better target segmentation effect; in addition, the second loss function can also be any other loss function that is beneficial to image segmentation and can be supervised by the true value.

[0017] In addition, when training the neural network, only the third target prediction feature map and the second target prediction feature map are used to calculate the second loss function, and the final output target object segmentation mask is not used. The reason for this is that the final output target object segmentation mask has a large amount of data after two upsamplings. In order to reduce the amount of calculation as much as possible based on the features of the context, the final output target object segmentation mask is not used. However, it should be noted that when the network training is completed and used, the final target object mask is the final output target object segmentation mask output by the first target prediction feature map, so that the segmented object is more accurate.

[0018] In a possible implementation, each feature prediction module includes a gating unit; the third residual feature map is input into the third feature prediction module to obtain a third target prediction feature map; the third target prediction feature map is upsampled and input into the second feature prediction module with the second residual feature map to obtain a second target prediction feature map; the upsampled third target prediction feature map and the second target prediction feature map are fused, and a second loss function is calculated based on the difference between the upsampled third target prediction feature map and the true segmentation mask, and also includes: the third residual feature map is input into the third gating unit in the third feature prediction module to obtain the third target prediction feature map; the third target prediction feature map and the second residual feature map are input into the second gating unit in the second feature prediction module to obtain the second target prediction feature map.

[0019] It should be noted that the gated unit can capture longer dependencies in time series data. Adding a gated recurrent unit to each feature prediction module can retain information for a longer period of time.

[0020] In a possible implementation, the image is input into a first residual calculation module to obtain a first residual feature map, the first residual feature map is input into a second residual calculation module to obtain a second residual feature map, and the second residual feature map is input into a third residual calculation module to obtain a third residual feature map, and also includes: the second target prediction feature map and the first residual feature map are input into a first feature prediction module to obtain a first target prediction feature map; the first residual calculation module includes residual calculation and a relu activation layer, and the first target prediction feature map is subtracted from the input image to obtain a first residual calculation result; and the first residual calculation result is input into the relu activation layer of the first residual calculation module to obtain the first residual feature map; the second residual calculation module includes a convolution layer, a pooling layer, a residual calculation, and a relu activation layer, and the second target prediction feature map is input into a 1*1 convolution layer. The invention relates to a method for extracting features through a convolution layer; subtracting a second target prediction feature map after feature extraction through a 1*1 convolution layer from a feature map after the first residual feature map passes through a convolution layer and a pooling layer of a second residual calculation module to obtain a second residual calculation result; inputting the second residual calculation result into a relu activation layer of the second residual calculation module to obtain the second residual feature map; the third residual calculation module comprises a convolution layer, a pooling layer, a residual calculation, and a relu activation layer, and inputting the third target prediction feature map into a 1*1 convolution layer for feature extraction; subtracting a third target prediction feature map after feature extraction through a 1*1 convolution layer from a feature map after the second residual feature map passes through a convolution layer and a pooling layer of a third residual calculation module to obtain a third residual calculation result; inputting the third residual calculation result into the relu activation layer of the third residual calculation module to obtain the third residual feature map.

[0021] It should be noted that the first residual calculation module is different from the second and third residual calculation modules. The first residual calculation module does not include a convolution layer and a pooling layer. Since it is in the input part of the convolutional neural network, it does not need to perform other feature extraction to obtain an image that meets the size requirements; thereby adjusting the image size.

[0022] In a possible implementation, the target person's posture detection is performed through the target person's mask to obtain the target person's step posture, including: determining the posture of the target person's mask through the translation and rotation of the object shape closest to the target person's mask:

[0023] t(R,T)=argmaxf(Mt,Mn)

[0024] Among them, t(R,T) is the posture of the target person, including the rotation R and translation T relative to the original position, f(Mt,Mn) represents the comparison function for measuring the overlapping area of ​​two masks; Mt represents the target person mask, Mn represents the target person mask set of different candidate postures; argmax represents the target person mask of the candidate posture with the largest overlapping area, and the posture of the target person mask is used as the target person step posture.

[0025] In 9D interaction, the amount of data processing is huge. In order to carry out real-time interaction, target objects with different postures are prepared in advance as a target object mask set with candidate postures. When detecting the posture of the target object, the posture of the target object can be selected by measuring the overlapping area, which reduces the amount of calculation.

[0026] In one possible implementation, controlling the second floor tile in the (N-2)×(N-2) sub-block areas to move in the first direction according to the first movement direction and the movement speed includes: determining a straight line according to the center point of the first area and the first movement direction; under the condition that the straight line is parallel to the rows or columns of the N×N sub-block areas: determining that the area in the (N-2)×(N-2) sub-block areas penetrated by the straight line is the second floor tile; and controlling the second floor tile to move in the first direction at the movement speed.

[0027] In a possible implementation, after determining the first floor tile according to the first movement direction and controlling the first floor tile to move to the empty area, the method further includes: determining a third floor tile penetrated by the straight line, the third floor tile and the first floor tile being symmetrical with the central sub-block; determining a fourth floor tile on one side of the outermost circle of the N×N sub-block area divided by the straight line; the fourth floor tile is a floor tile that connects the first floor tile and the third floor tile in the initial state to form the outermost side of the N×N sub-block area; and moving the third floor tile and the fourth floor tile clockwise or counterclockwise until the position of the first floor tile in the initial state is filled.

[0028] On the other hand, the present invention provides a 9D interactive system, which includes a camera, a projection device, a stereo audio device, a fragrance dissemination device, an environmental control device, a blowing device, and a vibration feedback device; the 9D interaction is deployed in a first area, the first area is a square, the first area is composed of N×N sub-block areas, N is a positive odd number and N≥3, each sub-block area is the same size as a floor tile, the number of floor tiles contained in the first area is less than N×N, and the 9D interactive system can control the floor tiles to move to any one of the sub-block areas in the first area; in an initial state, the outermost circle of the N×N sub-block areas is provided with floor tiles; the outermost circle of the (N-1)×(N-1) sub-block areas centered on the central sub-block is an empty area; the (N-2)×(N-2) sub-block areas centered on the central sub-block are all provided with floor tiles; the target person is located in the central sub-block in the initial state. Block, the 9D interactive system is used to execute the following method steps: a naked-eye 3D scene projected by the projection device; obtaining video data of the target person; extracting posture features of the target person in the video data; calculating the movement speed and the first movement direction of the target person according to the posture features; determining a first floor tile according to the first movement direction, and controlling the first floor tile to move to the empty area, wherein the first floor tile is the outermost circle of the N×N sub-block area in the initial state; controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in a first direction according to the first movement direction and the movement speed, wherein the first movement direction is opposite to the first direction; adjusting the picture projected by the projection device according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point of the first area, and the movement direction is the first direction.

[0029] On the other hand, the present invention provides an electronic device, comprising: a processor, suitable for implementing one or more instructions; and a computer storage medium, the computer storage medium storing one or more instructions, the one or more instructions being suitable for being loaded by the processor and executing the above-mentioned display method based on the motion state of a character.

[0030] On the other hand, the present invention provides a computer storage medium, characterized in that the computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the above-mentioned display method based on the motion state of a character.

[0031] It can be seen that the present invention can obtain a naked-eye 3D scene through projection of a projection device; obtain video data of a target person; extract the posture features of the target person in the video data; calculate the movement speed and the first movement direction of the target person according to the posture features; determine the first floor tile according to the first movement direction, and control the first floor tile to move to an empty area; control the second floor tile in (N-2)×(N-2) sub-block areas to move in the first direction according to the first movement direction and the movement speed, and the first movement direction is opposite to the first direction; adjust the picture projected by the projection device according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point in the first area, and the movement direction is the first direction. The range of the interactive scene is increased by moving the floor tiles, and the user's immersive experience is enhanced.

[0032] On the other hand, when identifying the posture features of the target person, the person mask is obtained first, and then the posture detection is performed based on the person mask. It is not necessary to process the video directly. Instead, the target person is hit in the image area of ​​the video and then the posture detection is performed through matching. The residual module and the target prediction module are used in the stage of obtaining the person mask to reduce the amount of data processing as much as possible while utilizing contextual features. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0034] Figure 1A It is a scenario architecture diagram provided by an embodiment of the present invention;

[0035] Figure 1B This is another scenario architecture diagram provided by an embodiment of the present invention;

[0036] Figure 1C This is another scenario architecture diagram provided by an embodiment of the present invention;

[0037] Figure 2 It is a flow chart of a display method based on the motion state of a person provided by an embodiment of the present invention;

[0038] Figure 3 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0040] like Figure 1A As shown, Figure 1A 1 is a scene architecture diagram provided in an embodiment of the present application. A display method based on a character's motion state provided in an embodiment of the present application can be applied to the scene architecture shown in FIG. 1 . Figure 1A The first area is specifically shown in the figure, where the first area consists of N×N sub-block areas, and N=9; the outermost circle of the first area is a movable floor tile, and the central 5×5 area is also a movable floor tile; the final floor tile is the central sub-block, and the outermost circle of the 6×6 area centered on the central sub-block is an empty area; in the initial state, the target person is located in the central sub-block. Figure 1B , when the target person moves along the Y axis, a straight line is determined by the moving direction and the center point of the first area, and the floor tiles that the straight line passes through are determined. Floor tile A1 is moved along the negative direction of the Y axis so that it is close to floor tile A2. At the same time, A2, A3, the central sub-square, A4 and A5 are moved along the negative direction of the Y axis, and the speed of A2, A3, the central sub-square, A4 and A5 moving along the negative direction of the Y axis is the same as the speed of the target person moving along the positive direction of the Y axis, thereby ensuring that the actual position of the target person in the first area does not change. See Figure 1C , determine a straight line in the Y-axis direction that passes through the center point of the first area, the first area is divided into two parts, and for the side of the positive X-axis in the two parts, the outermost floor tiles in the first area are moved counterclockwise to fill the position where the floor tile A1 is located in the initial state; by repeating the above steps, the arrangement of the floor tiles can be continuously restored to the arrangement in the initial state during the movement of the target person.

[0041] Based on the above description, an embodiment of the present invention proposes a display method based on the motion state of a character, which can be executed by an electronic device. The method is applied to a 9D interactive system, which includes a camera, a projection device, a stereo audio device, a fragrance propagation device, an environment control device, a blowing device, and a vibration feedback device;

[0042] The 9D interactive system is deployed in a first area, the first area is a square, the first area is composed of N×N sub-square areas, N is a positive odd number and N≥3, each sub-square area is the same size as a floor tile, the number of floor tiles included in the first area is less than N×N, and the 9D interactive system can control the floor tile to move to any of the sub-square areas in the first area;

[0043] In the initial state, the outermost circle of the N×N sub-square areas is provided with floor tiles; the outermost circle of the (N-1)×(N-1) sub-square areas centered on the central sub-square is an empty area; the (N-2)×(N-2) sub-square areas centered on the central sub-square are all provided with floor tiles; the target person is located in the central sub-square in the initial state;

[0044] See also Figure 2 The display method based on the character motion state may include the following steps S201-S205:

[0045] S201: Obtaining the naked-eye 3D scene projected by the projection device; acquiring the video data of the target person;

[0046] S202: extracting posture features of the target person in the video data; and calculating the movement speed and the first movement direction of the target person according to the posture features;

[0047] S203: determining a first floor tile according to the first movement direction, and controlling the first floor tile to move to the empty area, wherein the first floor tile is the floor tile of the outermost circle of the N×N sub-block area in the initial state;

[0048] S204: controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in a first direction according to the first movement direction and the movement speed, wherein the first movement direction is opposite to the first direction;

[0049] S205: Adjust the picture projected by the projection device according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point in the first area, and the movement direction is the first direction.

[0050] The movement direction of the floor tiles is the same as the movement direction of the target person, and they maintain the same speed; the display of the naked-eye 3D scene is synchronized with the movement direction and speed of the target person. For example, if the target person moves along the positive direction of the Y axis, the direction of movement of the floor tiles is the negative direction of the Y axis; and the 3D scene can calculate the distance and position of the target person in the 3D scene based on the current moving speed and direction of the target person, and then display the corresponding 3D scene in real time based on the calculated position. Through the coordination of the floor tiles and the 3D scene, when the target person moves forward, the actual position of the target person in space does not change, but the 3D scene at that position does change based on the moving speed of the person, giving the user an immersive feeling that he or she is actually moving forward.

[0051] For example, in the initial state, the position of the target person corresponds to position A in the 3D scene, and the perspective scene corresponding to position A is projected in space. The 9D interactive system can calculate the distance based on the target person's moving speed, moving direction and time, and determine position B in the 3D scene. The 9D interactive system then projects the perspective scene corresponding to position B in space.

[0052] It can be seen that in the embodiment of the present application, the floor tiles move in the opposite direction to the personnel, so that the target person is always in the central area of ​​the space within the limited space, thereby broadening the scope of the 9D interactive scene and enhancing the user's immersive experience.

[0053] It should be noted that the 9D interactive system can determine the moving distance and moving time of the human body based on the time interval between adjacent video frames and the posture features between adjacent video frames, and then calculate the movement speed and the first movement direction.

[0054] Optionally, 9D refers to naked eye 3D, vibration, floor tiles moving forward and backward, left and right, rotation, smell, wind, rain and snow.

[0055] It can be seen that the present invention can obtain a naked-eye 3D scene through projection of a projection device; obtain video data of a target person; extract the posture features of the target person in the video data; calculate the movement speed and the first movement direction of the target person according to the posture features; determine the first floor tile according to the first movement direction, and control the first floor tile to move to an empty area; control the second floor tile in (N-2)×(N-2) sub-block areas to move in the first direction according to the first movement direction and the movement speed, and the first movement direction is opposite to the first direction; adjust the picture projected by the projection device according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point in the first area, and the movement direction is the first direction. The range of the interactive scene is increased by moving the floor tiles, and the user's immersive experience is enhanced.

[0056] In a possible implementation, the extraction of the posture features of the target person in the video data; the calculation of the movement speed and the first movement direction of the target person according to the posture features, including: obtaining an input image according to the video data, inputting the image into an image segmentation model for image segmentation, and obtaining a target person mask; performing target posture detection through the target person mask to obtain the target person's step posture; determining the movement speed and the first movement direction of the target person according to the target person's step posture; wherein the image segmentation model is composed of three layers, the first layer includes a first feature prediction module and a first residual calculation module, the second layer includes a second feature prediction module and a second residual calculation module, and the third layer includes a third feature prediction module and a third residual calculation module; the first residual calculation module, the second residual module and the third residual module are connected in sequence, and the residual feature map of the layer is extracted; the third feature prediction module, the second feature prediction module and the first feature prediction module are connected in sequence, and the target prediction feature map of the layer is obtained, wherein the output of the first feature prediction module as a target person mask; the training method of the image segmentation model is: inputting the image into the first residual calculation module to obtain a first residual feature map, inputting the first residual feature map into the second residual calculation module to obtain a second residual feature map, and inputting the second residual feature map into the third residual calculation module to obtain a third residual feature map; calculating the sum of the first residual feature map, the second residual feature map and the third residual feature map, and calculating the self-supervised loss of the image segmentation model according to the sum of the above residual feature maps as the first loss function; inputting the third residual feature map into the third feature prediction module to obtain a third target prediction feature map; upsampling the third target prediction feature map and the second residual feature map into the second feature prediction module to obtain a second target prediction feature map; fusing the upsampled third target prediction feature map and the second target prediction feature map, and calculating the second loss function according to the difference between the upsampled third target prediction feature map and the real segmentation mask; adding the first loss function and the second loss function to obtain the target segmentation loss function; using the target segmentation loss function to train the initial segmentation module to obtain the image segmentation model.

[0057] It should be noted that the basic idea of ​​Dice loss is to calculate the overlap between the predicted results and the true results, and optimize the model by minimizing the difference between the two; Focal loss handles the class imbalance problem by assigning more weights to examples that are difficult to classify or easily misclassified and reducing the weights of simple examples; the second loss function can be calculated using the sum of Dice loss and Focal loss to achieve a better target segmentation effect; in addition, the second loss function can also be any other loss function that is beneficial to image segmentation and can be supervised by the true value.

[0058] In addition, when training the neural network, only the third target prediction feature map and the second target prediction feature map are used to calculate the second loss function, and the final output target object segmentation mask is not used. The reason for this is that the final output target object segmentation mask has a large amount of data after two upsamplings. In order to reduce the amount of calculation as much as possible based on the features of the context, the final output target object segmentation mask is not used. However, it should be noted that when the network training is completed and used, the final target object mask is the final output target object segmentation mask output by the first target prediction feature map, so that the segmented object is more accurate.

[0059] In a possible implementation, each feature prediction module includes a gating unit; the third residual feature map is input into the third feature prediction module to obtain a third target prediction feature map; the third target prediction feature map is upsampled and input into the second feature prediction module with the second residual feature map to obtain a second target prediction feature map; the upsampled third target prediction feature map and the second target prediction feature map are fused, and a second loss function is calculated based on the difference between the upsampled third target prediction feature map and the true segmentation mask, and also includes: the third residual feature map is input into the third gating unit in the third feature prediction module to obtain the third target prediction feature map; the third target prediction feature map and the second residual feature map are input into the second gating unit in the second feature prediction module to obtain the second target prediction feature map.

[0060] It should be noted that the gated unit can capture longer dependencies in time series data. Adding a gated recurrent unit to each feature prediction module can retain information for a longer period of time.

[0061] In a possible implementation, the image is input into a first residual calculation module to obtain a first residual feature map, the first residual feature map is input into a second residual calculation module to obtain a second residual feature map, and the second residual feature map is input into a third residual calculation module to obtain a third residual feature map, and also includes: the second target prediction feature map and the first residual feature map are input into a first feature prediction module to obtain a first target prediction feature map; the first residual calculation module includes residual calculation and a relu activation layer, and the first target prediction feature map is subtracted from the input image to obtain a first residual calculation result; and the first residual calculation result is input into the relu activation layer of the first residual calculation module to obtain the first residual feature map; the second residual calculation module includes a convolution layer, a pooling layer, a residual calculation, and a relu activation layer, and the second target prediction feature map is input into a 1*1 convolution layer. The invention relates to a method for extracting features through a convolution layer; subtracting a second target prediction feature map after feature extraction through a 1*1 convolution layer from a feature map after the first residual feature map passes through a convolution layer and a pooling layer of a second residual calculation module to obtain a second residual calculation result; inputting the second residual calculation result into a relu activation layer of the second residual calculation module to obtain the second residual feature map; the third residual calculation module comprises a convolution layer, a pooling layer, a residual calculation, and a relu activation layer, and inputting the third target prediction feature map into a 1*1 convolution layer for feature extraction; subtracting a third target prediction feature map after feature extraction through a 1*1 convolution layer from a feature map after the second residual feature map passes through a convolution layer and a pooling layer of a third residual calculation module to obtain a third residual calculation result; inputting the third residual calculation result into the relu activation layer of the third residual calculation module to obtain the third residual feature map.

[0062] It should be noted that the first residual calculation module is different from the second and third residual calculation modules. The first residual calculation module does not include a convolution layer and a pooling layer. Since it is in the input part of the convolutional neural network, it does not need to perform other feature extraction to obtain an image that meets the size requirements; thereby adjusting the image size.

[0063] In a possible implementation, the target person's posture detection is performed through the target person's mask to obtain the target person's step posture, including: determining the posture of the target person's mask through the translation and rotation of the object shape closest to the target person's mask:

[0064] t(R,T)=argmaxf(Mt,Mn)

[0065] Among them, t(R,T) is the posture of the target person, including the rotation R and translation T relative to the original position, f(Mt,Mn) represents the comparison function for measuring the overlapping area of ​​two masks; Mt represents the target person mask, Mn represents the target person mask set of different candidate postures; argmax represents the target person mask of the candidate posture with the largest overlapping area, and the posture of the target person mask is used as the target person step posture.

[0066] It should be noted that the amount of data processing in 9D interaction is huge. In order to carry out real-time interaction, the target objects with different postures are made into a target object mask set of candidate postures in advance; when detecting the posture of the target object, it is only necessary to select the posture of the target object by measuring the overlapping area, thereby reducing the amount of calculation.

[0067] In one possible implementation, controlling the second floor tile in the (N-2)×(N-2) sub-block areas to move in the first direction according to the first movement direction and the movement speed includes: determining a straight line according to the center point of the first area and the first movement direction; under the condition that the straight line is parallel to the rows or columns of the N×N sub-block areas: determining that the area in the (N-2)×(N-2) sub-block areas penetrated by the straight line is the second floor tile; and controlling the second floor tile to move in the first direction at the movement speed.

[0068] For example, see Figure 1B , N=9, then in the central 5×5 area, the floor tiles A2, A3, central sub-squares, A4 and A5 penetrated by the straight line are the second floor tiles; the first direction is the negative direction of the Y axis, then the floor tiles A2, A3, central sub-squares, A4 and A5 are moved along the negative direction of the Y axis.

[0069] In a possible implementation, after determining the first floor tile according to the first movement direction and controlling the first floor tile to move to the empty area, the method further includes: determining a third floor tile penetrated by the straight line, the third floor tile and the first floor tile being symmetrical with the central sub-block; determining a fourth floor tile on one side of the outermost circle of the N×N sub-block area divided by the straight line; the fourth floor tile is a floor tile that connects the first floor tile and the third floor tile in the initial state to form the outermost side of the N×N sub-block area; and moving the third floor tile and the fourth floor tile clockwise or counterclockwise until the position of the first floor tile in the initial state is filled.

[0070] For example, see Figure 1C, the floor tile below the floor tile A5, that is, the floor tile in the negative direction of the Y axis, is the third floor tile. The third floor tile and the floor tile in the positive direction of the straight line X axis are moved counterclockwise to fill the position of the floor tile A1 in the initial state, so that the position where the third floor tile is located in the initial state is vacant and filled by the floor tile A5. The above steps are repeated to enable the arrangement of the floor tiles to be continuously restored to the arrangement in the initial state during the target person's movement along the positive direction of the Y axis.

[0071] On the other hand, an embodiment of the present invention provides a 9D interactive system, the 9D interactive system comprising a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environmental control device, a blowing device, and a vibration feedback device; the 9D interaction is deployed in a first area, the first area is a square, the first area is composed of N×N sub-block areas, N is a positive odd number and N≥3, each sub-block area is the same size as a floor tile, the number of floor tiles contained in the first area is less than N×N, and the 9D interactive system can control the floor tiles to move to any one of the sub-block areas in the first area; in an initial state, the outermost circle of the N×N sub-block areas is provided with floor tiles; the outermost circle of the (N-1)×(N-1) sub-block areas centered on the central sub-block is an empty area; the (N-2)×(N-2) sub-block areas centered on the central sub-block are all provided with floor tiles; the target person is located in the central sub-block in the initial state, and the 9D interactive system is used to perform the following method steps:

[0072] The naked eye 3D scene projected by the projection device;

[0073] Acquire video data of the target person;

[0074] Extracting posture features of the target person in the video data;

[0075] Calculate the movement speed and the first movement direction of the target person according to the posture feature;

[0076] Determine a first floor tile according to the first movement direction, control the first floor tile to move to the empty area, the first floor tile being the floor tile of the outermost circle of the N×N sub-block area in the initial state;

[0077] Controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in a first direction according to the first movement direction and the movement speed, wherein the first movement direction is opposite to the first direction;

[0078] The picture projected by the projection device is adjusted according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point in the first area, and the movement direction is the first direction.

[0079] The embodiment of the present invention also provides a computer storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer storage medium here can include both built-in storage media in the electronic device and, of course, extended storage media supported by the electronic device. The computer storage medium provides a storage space, which stores the operating system of the electronic device. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0080] Figure 3 3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The device includes: at least one processor 301, such as a central processing unit (CPU), at least one memory 302, and at least one bus 303.

[0081] The memory 302 may store program instructions, and the processor 301 may be used to call the program instructions to execute a display system and method based on the motion state of a person.

[0082] It should be noted that all steps of the embodiments of the present application are performed under legal and compliant conditions, that is, all steps of the embodiments of the present application are performed with authorization.

[0083] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A display method based on the motion state of a person, characterized in that: The method is applied to a 9D interactive system, which includes a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environment control device, a blowing device, and a vibration feedback device; The 9D interactive system is deployed in a first area, the first area is a square, the first area is composed of N×N sub-square areas, N is a positive odd number and N≥3, each sub-square area is the same size as a floor tile, the number of floor tiles included in the first area is less than N×N, and the 9D interactive system can control the floor tile to move to any of the sub-square areas in the first area; In the initial state, the outermost circle of the N×N sub-square areas is set with floor tiles; the outermost circle of the (N-1)×(N-1) sub-square areas centered on the central sub-square is an empty area; the (N-2)×(N-2) sub-square areas centered on the central sub-square are all set with floor tiles; The target person is located in the central sub-square in the initial state; the method comprises: The naked eye 3D scene projected by the projection device; Acquire video data of the target person; Extracting posture features of the target person in the video data; Calculate the movement speed and the first movement direction of the target person according to the posture feature; Determine a first floor tile according to the first movement direction, control the first floor tile to move in the first direction to the empty area, the first floor tile being the floor tile of the outermost circle of the N×N sub-block area in the initial state; Controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in a first direction according to the first movement direction and the movement speed, wherein the first movement direction is opposite to the first direction; Adjusting the display image projected by the projection device according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point in the first area, and the movement direction is the first direction; The step of controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in the first direction according to the first movement direction and the movement speed comprises: Determine a straight line in a first direction of motion that passes through a center point of the first region; Determine the sub-square area among the (N-2)×(N-2) sub-square areas that is penetrated by the straight line as the second floor tile; Controlling the second floor tile to move toward the first direction at the movement speed; After determining the first floor tile according to the first movement direction and controlling the first floor tile to move to the empty area, the method further includes: Determine a third floor tile that is penetrated by the straight line, wherein the third floor tile is symmetrical to the first floor tile with respect to the central sub-square; Determine a fourth floor tile on one side of a circle of the outermost part of the N×N sub-block area divided by the straight line; the fourth floor tile is a floor tile that connects the first floor tile and the third floor tile in the initial state to form the outermost side of the N×N sub-block area; The third floor tile and the fourth floor tile are moved clockwise or counterclockwise until the position where the first floor tile is located in the initial state is filled.

2. The method according to claim 1, characterized in that Extracting posture features of the target person in the video data; Calculating the movement speed and the first movement direction of the target person according to the posture feature includes: Acquire an input image according to the video data, input the image into an image segmentation model for image segmentation, and obtain a target person mask; Detect the target person's posture through the target person's mask to obtain the target person's step posture; Determine the moving speed and the first moving direction of the target person according to the step posture of the target person; The image segmentation model is composed of three layers, the first layer includes a first feature prediction module and a first residual calculation module, the second layer includes a second feature prediction module and a second residual calculation module, and the third layer includes a third feature prediction module and a third residual calculation module; the first residual calculation module, the second residual calculation module and the third residual calculation module are connected in sequence, and the residual feature map of the layer is extracted; the third feature prediction module, the second feature prediction module and the first feature prediction module are connected in sequence, and the target prediction feature map of the layer is obtained, wherein the output of the first feature prediction module is used as the target person mask; The training method of the image segmentation model is: Inputting the image into a first residual calculation module to obtain a first residual feature map, inputting the first residual feature map into a second residual calculation module to obtain a second residual feature map, and inputting the second residual feature map into a third residual calculation module to obtain a third residual feature map; Calculating the sum of the first residual feature map, the second residual feature map and the third residual feature map, and calculating the self-supervised loss of the image segmentation model according to the sum of the residual feature maps as a first loss function; Inputting the third residual feature map into a third feature prediction module to obtain a third target prediction feature map; inputting the third target prediction feature map and the second residual feature map into a second feature prediction module after upsampling to obtain a second target prediction feature map; fusing the upsampled third target prediction feature map with the second target prediction feature map, and calculating a second loss function based on the difference between the third target prediction feature map and the true segmentation mask; Adding the first loss function and the second loss function to obtain a target segmentation loss function; The target segmentation loss function is used to train the initial segmentation module to obtain the image segmentation model.

3. The method according to claim 2, characterized in that Each feature prediction module includes a gating unit; the third residual feature map is input into the third feature prediction module to obtain a third target prediction feature map; the third target prediction feature map is upsampled and input into the second feature prediction module with the second residual feature map to obtain a second target prediction feature map; the upsampled third target prediction feature map is fused with the second target prediction feature map, and a second loss function is calculated according to the difference between the upsampled third target prediction feature map and the true segmentation mask, and further includes: Inputting the third residual feature map into a third gating unit in a third feature prediction module to obtain the third target prediction feature map; The third target prediction feature map and the second residual feature map are input into a second gating unit in a second feature prediction module to obtain the second target prediction feature map.

4. The method according to claim 3, characterized in that The method of inputting the image into a first residual calculation module to obtain a first residual feature map, inputting the first residual feature map into a second residual calculation module to obtain a second residual feature map, and inputting the second residual feature map into a third residual calculation module to obtain a third residual feature map further includes: The second target prediction feature map and the first residual feature map are input into a first feature prediction module to obtain a first target prediction feature map; The first residual calculation module includes residual calculation and a relu activation layer, and the first target prediction feature map is subtracted from the input image to obtain a first residual calculation result; and the first residual calculation result is input into the relu activation layer of the first residual calculation module to obtain the first residual feature map; The second residual calculation module includes a convolution layer, a pooling layer, a residual calculation, and a relu activation layer. The second target prediction feature map is input into a 1*1 convolution layer for feature extraction; the second target prediction feature map after feature extraction by the 1*1 convolution layer is subtracted from the feature map after the first residual feature map passes through the convolution layer and the pooling layer of the second residual calculation module to obtain a second residual calculation result; the second residual calculation result is input into the relu activation layer of the second residual calculation module to obtain the second residual feature map; The third residual calculation module includes a convolution layer, a pooling layer, a residual calculation, and a relu activation layer. The third target prediction feature map is input into a 1*1 convolution layer for feature extraction; the third target prediction feature map after feature extraction by the 1*1 convolution layer is subtracted from the feature map of the second residual feature map after the convolution layer and the pooling layer of the third residual calculation module to obtain a third residual calculation result; the third residual calculation result is input into the relu activation layer of the third residual calculation module to obtain the third residual feature map.

5. The method according to claim 2, characterized in that The target person's posture detection is performed through the target person's mask to obtain the target person's step posture, including: t(R,T)=argmaxf(Mt,Mn) Among them, t(R,T) is the posture of the target person, including the rotation and translation relative to the original position, f(Mt, Mn) represents the comparison function for measuring the overlapping area of ​​two masks; Mt represents the target person mask, Mn represents the target person mask set of different candidate postures; argmax represents the target person mask of the candidate posture with the largest overlapping area, and the posture of the target person mask is used as the target person step posture.

6. A 9D interactive system, characterized in that: The 9D interactive system includes a camera, a projection device, a stereo audio device, a fragrance diffusion device, an environmental control device, a hair blowing device, and a vibration feedback device; the 9D interactive system is deployed in a first area, the first area is a square, the first area is composed of N×N sub-block areas, N is a positive odd number and N≥3, each sub-block area is the same size as a floor tile, the number of floor tiles contained in the first area is less than N×N, and the 9D interactive system can control the floor tiles to move to any of the sub-block areas in the first area; in an initial state, the outermost circle of the N×N sub-block areas is provided with floor tiles; the outermost circle of the (N-1)×(N-1) sub-block areas centered on the central sub-block is an empty area; the (N-2)×(N-2) sub-block areas centered on the central sub-block are all provided with floor tiles; the target person is located in the central sub-block in the initial state, and the 9D interactive system is used to perform the following method steps: The naked eye 3D scene projected by the projection device; Acquire video data of the target person; Extracting posture features of the target person in the video data; Calculate the movement speed and the first movement direction of the target person according to the posture feature; Determine a first floor tile according to the first movement direction, control the first floor tile to move in the first direction to the empty area, the first floor tile being the floor tile of the outermost circle of the N×N sub-block area in the initial state; Controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in a first direction according to the first movement direction and the movement speed, wherein the first movement direction is opposite to the first direction; Adjust the picture projected by the projection device according to the first movement direction and the movement speed, so that the naked-eye 3D scene moves at the movement speed relative to the center point in the first area, and the movement direction is the first direction; The step of controlling the second floor tile in the (N-2)×(N-2) sub-block area to move in the first direction according to the first movement direction and the movement speed comprises: Determine a straight line in a first direction of motion that passes through a center point of the first region; Determine the sub-square area among the (N-2)×(N-2) sub-square areas that is penetrated by the straight line as the second floor tile; Controlling the second floor tile to move toward the first direction at the movement speed; After determining the first floor tile according to the first movement direction and controlling the first floor tile to move to the empty area, the method further includes: Determine a third floor tile that is penetrated by the straight line, wherein the third floor tile is symmetrical to the first floor tile with respect to the central sub-square; Determine a fourth floor tile on one side of a circle of the outermost part of the N×N sub-block area divided by the straight line; the fourth floor tile is a floor tile that connects the first floor tile and the third floor tile in the initial state to form the outermost side of the N×N sub-block area; The third floor tile and the fourth floor tile are moved clockwise or counterclockwise until the position where the first floor tile is located in the initial state is filled.

7. An electronic device, characterized in that: include: a processor adapted to implement one or more instructions; as well as, A computer storage medium storing one or more instructions, wherein the one or more instructions are suitable for being loaded by the processor and executing a display method based on a character's motion state as described in any one of claims 1-5.

8. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing a display method based on a character's motion state as described in any one of claims 1-5.

Images