Interactive drawing table control method based on human body posture recognition and interactive drawing table

Abstract

The invention relates to the technical field of computers, and discloses an interactive drawing table control method based on human body posture recognition and an interactive drawing table, and the method comprises the steps: obtaining initial user images of a to-be-monitored user at a plurality of continuous moments based on a visual camera; inputting the initial user image into an image processing model to obtain a target user image of the to-be-monitored user output by the image processing model; inputting the target user image into a posture recognition model to obtain a user posture recognition result of the to-be-monitored user output by the posture recognition model; and displaying corresponding posture correction information on an interactive display interface of the interactive drawing table based on the user posture recognition result, so that the to-be-monitored user performs posture correction according to the posture correction information. According to the invention, the recognition accuracy of the user posture can be improved, and the control accuracy of the interactive drawing table based on human body posture recognition is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an interactive drawing table control method based on human body posture recognition and an interactive drawing table. Background Art

[0002] With the development of artificial intelligence technology, smart interactive drawing tables have been gradually promoted. The essence of interactive drawing table control is to monitor various situations through video, identify abnormal behaviors based on video monitoring, and prompt alarm information after identifying abnormal behaviors, and users can correct their behaviors according to the alarm information.

[0003] Existing methods generally use target detection to detect abnormal behaviors, and specifically, they generally use typical target detection algorithms to detect abnormal behaviors. In existing methods, abnormal behavior detection directly based on algorithms is limited by the scene and angle, and is prone to false detection, resulting in low detection accuracy, which in turn leads to low accuracy in interactive drawing table control. Summary of the invention

[0004] The present invention provides an interactive drawing table control method based on human body posture recognition and an interactive drawing table, which are used to improve the recognition accuracy of user posture and further improve the accuracy of interactive drawing table control based on human body posture recognition.

[0005] In a first aspect, the present invention provides an interactive drawing table control method based on human posture recognition, which is applied to an interactive drawing table, wherein the interactive drawing table is equipped with a visual camera. The interactive drawing table control method based on human posture recognition comprises:

[0006] Acquire initial user images of the user to be monitored at multiple consecutive moments based on the visual camera;

[0007] Inputting the initial user image into an image processing model to obtain a target user image of the user to be monitored output by the image processing model;

[0008] Inputting the target user image into a gesture recognition model to obtain a user gesture recognition result of the user to be monitored output by the gesture recognition model; the gesture recognition model is trained based on sample user images and user gesture labels corresponding to the sample user images;

[0009] Based on the user posture recognition result, corresponding posture correction information is displayed on the interactive display interface of the interactive drawing table, so that the user to be monitored performs posture correction according to the posture correction information;

[0010] Among them, the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

[0011] According to the interactive drawing table control method based on human posture recognition provided by an embodiment of the present invention, the initial user image is input into the image processing model to obtain the target user image of the user to be monitored output by the image processing model, including:

[0012] Inputting the initial user image into a first feature extraction layer in the image processing model to obtain an initial feature map output by the first feature extraction layer;

[0013] Inputting the initial feature map into the second feature extraction layer in the image processing model to obtain at least one hierarchical feature map output by the second feature extraction layer; the scale of the at least one hierarchical feature map is smaller than the preset initial scale, and the scales of the at least one hierarchical feature map are different;

[0014] Inputting the initial feature map and the at least one hierarchical feature map into a feature fusion layer in the image processing model to obtain a target user image output by the feature fusion layer;

[0015] The feature fusion layer is used to upsample the input feature maps of different scales into upsampled feature maps with a scale of the preset initial scale, and perform feature fusion on each of the upsampled feature maps to obtain the target user image.

[0016] According to the interactive drawing table control method based on human posture recognition provided by an embodiment of the present invention, any of the hierarchical feature maps is extracted based on the hierarchical feature extraction layer in the second feature extraction layer, and the hierarchical feature map is extracted based on the following method:

[0017] Inputting the initial feature map input to the hierarchical feature extraction layer to the downsampling layer in the hierarchical feature extraction layer to obtain the downsampling feature map output by the downsampling layer; the downsampling layer is used to segment the initial feature map and splice the segmented feature maps in the channel dimension to reduce the scale of the initial feature map and increase the number of channels of the initial feature map;

[0018] The downsampled feature map is input to a third feature extraction layer in the hierarchical feature extraction layer to obtain the hierarchical feature map output by the third feature extraction layer; the scale of the hierarchical feature map is the same as the scale of the downsampled feature map.

[0019] According to the interactive drawing table control method based on human posture recognition provided by an embodiment of the present invention, the number of channels of the initial feature map is a preset initial channel number;

[0020] The step of inputting the initial feature map and the at least one hierarchical feature map into a feature fusion layer in the image processing model to obtain a target user image output by the feature fusion layer includes:

[0021] Inputting the initial feature map and the at least one hierarchical feature map into the channel number unification layer in the feature fusion layer, obtaining the initial feature map with unified channel numbers and the at least one hierarchical feature map with unified channel numbers output by the channel number unification layer; the channel number unification layer is used to convert the input feature maps with different channel numbers into feature maps with the preset initial channel number;

[0022] Inputting the initial feature map with a unified number of channels and at least one hierarchical feature map with a unified number of channels into the upsampling layer in the feature fusion layer to obtain a plurality of upsampling feature maps output by the upsampling layer;

[0023] Each of the upsampled feature maps is input into the feature splicing layer in the feature fusion layer to obtain a target user image output by the feature splicing layer.

[0024] According to the interactive drawing table control method based on human posture recognition provided by an embodiment of the present invention, the step of inputting the downsampled feature map to the third feature extraction layer in the hierarchical feature extraction layer to obtain the hierarchical feature map output by the third feature extraction layer includes:

[0025] Inputting the downsampled feature map into the first self-attention layer in the third feature extraction layer to obtain a first feature map output by the first self-attention layer; the first self-attention layer is used to divide the downsampled feature map into multiple windows for self-attention processing respectively;

[0026] The first feature map is input into the second self-attention layer in the third feature extraction layer to obtain the hierarchical feature map output by the second self-attention layer; the second self-attention layer is used to divide the first feature map into multiple windows for self-attention processing, and to transfer information between the multiple windows after the first feature map is divided.

[0027] According to the interactive drawing table control method based on human posture recognition provided by an embodiment of the present invention, the initial user image is input into the first feature extraction layer in the image processing model to obtain the initial feature map output by the first feature extraction layer, including:

[0028] Segmenting the initial user image into a plurality of image blocks;

[0029] Superimposing the multiple image blocks in the channel dimension to obtain an original feature map; the scale of the original feature map is the preset initial scale;

[0030] The original feature map is input into the first feature extraction layer to obtain an initial feature map output by the first feature extraction layer.

[0031] In a second aspect, the present invention provides an interactive drawing table, wherein the interactive drawing table is equipped with a visual camera, and the interactive drawing table comprises:

[0032] An acquisition module, used for acquiring initial user images of the user to be monitored at multiple consecutive moments based on the visual camera;

[0033] An image processing module, used for inputting the initial user image into an image processing model to obtain a target user image of the user to be monitored output by the image processing model;

[0034] A gesture recognition module, used to input the target user image into a gesture recognition model to obtain a user gesture recognition result of the user to be monitored output by the gesture recognition model; the gesture recognition model is trained based on sample user images and user gesture labels corresponding to the sample user images;

[0035] An interactive display module, configured to display corresponding posture correction information on an interactive display interface of the interactive drawing table based on the user posture recognition result, so that the user to be monitored can perform posture correction according to the posture correction information;

[0036] Among them, the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

[0037] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the interactive drawing table control method based on human posture recognition as described in the first aspect is implemented.

[0038] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the interactive drawing table control method based on human posture recognition as described in the first aspect.

[0039] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the interactive drawing table control method based on human posture recognition as described in the first aspect is implemented.

[0040] The interactive drawing table control method based on human posture recognition provided by the present invention processes the initial user image collected by the visual camera through the first feature extraction layer, the second feature extraction layer and the feature fusion layer in the image processing model, optimizes the image information in the initial user image, and further highlights the detail features in the target user image. The target user image is further subjected to posture recognition through a pre-trained posture recognition model to obtain an accurate user posture recognition result, and the interactive drawing table is further controlled based on the accurate user posture recognition result. Therefore, the recognition accuracy of the user posture can be improved through the artificial intelligence of the image processing model and the posture recognition model, thereby improving the accuracy of the interactive drawing table control based on human posture recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flow chart of the interactive drawing table control method based on human posture recognition provided by the present invention;

[0043] Figure 2 It is one of the structural schematic diagrams of the image processing model provided by the present invention;

[0044] Figure 3 This is the second structural schematic diagram of the image processing model provided by the present invention;

[0045] Figure 4 It is a structural schematic diagram of the interactive drawing table provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0048] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0049] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present invention.

[0050] See also Figure 1 , Figure 1 : is a flow chart of an interactive drawing table control method based on human posture recognition provided by the present invention. The method of the embodiment of the present invention is applied to an interactive drawing table, in which a visual camera is installed. The interactive drawing table control method based on human posture recognition includes:

[0051] Step 10: acquiring initial user images of the user to be monitored at multiple consecutive moments based on a visual camera.

[0052] The interactive drawing table in the embodiment of the present invention can be understood as a multifunctional intelligent desk. Therefore, after the interactive drawing table is started, a pop-up will appear on the interactive desktop of the interactive drawing table to ask whether the interactive drawing table needs to be controlled. After receiving a request to determine that the interactive drawing table needs to be controlled, the visual camera in the interactive drawing table is called to obtain the initial user images of the user to be monitored at multiple consecutive moments, wherein the user to be monitored is the user object in front of the interactive drawing table.

[0053] Furthermore, the collected initial user image is detected, and if it is detected that it does not meet the preset requirements, it is re-collected, thereby ensuring that the initial user image with clear shooting and good angle is input into the image processing model, thereby improving the accuracy of the interactive drawing table control.

[0054] Step 20: input the initial user image into the image processing model to obtain the target user image of the user to be monitored output by the image processing model.

[0055] Optionally, the initial user image is input into an image processing model to obtain a target user image of the user to be monitored output by the image processing model, wherein the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

[0056] Step 30: input the target user image into the gesture recognition model to obtain the user gesture recognition result of the user to be monitored output by the gesture recognition model.

[0057] Optionally, the target user image is input into a posture recognition model to obtain a user posture recognition result of the monitored user output by the posture recognition model, wherein the user posture recognition result may include, for example, the user's current posture is a tilted state, the user's current state is the body tilted 30 degrees to the left, the user's current state is the body tilted 15 degrees to the left, the user's current state is the body tilted 20 degrees to the right, etc.

[0058] The gesture recognition model in the embodiment of the present invention is trained based on the sample user images and the user gesture labels corresponding to the sample user images. Here, the sample initial user images are also images obtained by collecting images of the user. Further, the collected sample initial user images are cleaned to retain the sample initial user images with clear shooting and good angles, thereby improving the model training effect and further improving the accuracy of the interactive drawing table control.

[0059] For example, RGB images containing users are collected, the data is cleaned, invalid data is removed, and sample initial user images with clear normal images and complete users are selected. Further, training sets and test sets are constructed respectively to train the gesture recognition model.

[0060] Here, the user posture label is obtained by annotating the sample initial user image. For example, the user posture label can be standing, lying sideways, lying prone, sitting, etc. Further, the posture recognition model is trained based on the sample initial user image, at least one target detection frame corresponding to the sample initial user image, and the user posture label corresponding to each target detection frame; that is, the user in the sample initial user image is first annotated with a target detection frame, and the user posture label corresponding to the target detection frame is annotated, and the target detection frame can be a rectangular frame; that is, the sample initial user image is annotated with a position and a user posture label.

[0061] Step 40: display corresponding posture correction information on the interactive display interface of the interactive drawing table based on the user posture recognition result, so that the user to be monitored can perform posture correction according to the posture correction information.

[0062] Optionally, corresponding posture correction information is generated according to the user posture recognition result. For example, if the user posture recognition result is that the body is leaning to the left, the posture correction information is to return to the right.

[0063] Furthermore, the posture correction information is displayed on the interactive display interface of the interactive drawing table, so that the user to be monitored can perform posture correction according to the posture correction information.

[0064] The embodiment of the present invention processes the initial user image collected by the visual camera through the first feature extraction layer, the second feature extraction layer and the feature fusion layer in the image processing model, optimizes the image information in the initial user image, and further highlights the detail features in the target user image. The target user image is further subjected to posture recognition through a pre-trained posture recognition model to obtain an accurate user posture recognition result, and the interactive drawing table is further controlled based on the accurate user posture recognition result. Therefore, the recognition accuracy of the user posture can be improved through the artificial intelligence of the image processing model and the posture recognition model, thereby improving the accuracy of the interactive drawing table control based on human posture recognition.

[0065] In one embodiment, the initial user image is input into the image processing model to obtain the target user image of the user to be monitored output by the image processing model, including:

[0066] Step 201: input the initial user image into the first feature extraction layer in the image processing model to obtain an initial feature map output by the first feature extraction layer.

[0067] Optionally, the initial user image is preprocessed and then the preprocessed initial user image is input into the image processing model. The preprocessing may include but is not limited to at least one of the following: scale adjustment, normalization processing, etc.

[0068] Optionally, the initial user image is divided into multiple image blocks, and the multiple image blocks are superimposed in the channel dimension to obtain an original feature map, the scale of the original feature map is a preset initial scale, so that the original feature map is input into the first feature extraction layer, thereby extracting feature information at different positions in the initial user image, further improving the feature extraction capability, and then further improving the image feature representation capability, and ultimately improving the accuracy of the interactive drawing table control.

[0069] The first feature extraction layer is used to extract a feature map of a preset initial scale. That is, the first feature extraction layer is used to extract features from the initial user image to obtain an initial feature map of a preset initial scale. The preset initial scale can be set according to actual needs.

[0070] In one embodiment, the first feature extraction layer includes a linear embedding layer, and the initial user image is feature mapped to obtain an initial feature map. Optionally, the first feature extraction layer includes a linear embedding layer, a first self-attention layer, and a second self-attention layer connected in sequence; specifically, the initial user image is input into the linear embedding layer, and the feature map output by the linear embedding layer is input into the first self-attention layer to obtain a second feature map output by the first self-attention layer; the first self-attention layer is used to divide the input feature map into multiple windows for self-attention processing respectively; the second feature map is input into the second self-attention layer to obtain the initial feature map output by the second self-attention layer; the second self-attention layer is used to divide the second feature map into multiple windows for self-attention processing, and to enable information transfer between the multiple windows after the second feature map is divided.

[0071] Step 202: input the initial feature map to the second feature extraction layer in the image processing model to obtain at least one hierarchical feature map output by the second feature extraction layer.

[0072] Among them, the scale of at least one hierarchical feature map is smaller than the preset initial scale, and the scales of at least one hierarchical feature map are different. That is, the second feature extraction layer can continuously extract features of the initial feature map, and can extract hierarchical feature maps of different scales and levels, thereby further improving the feature extraction capability and ultimately improving the accuracy of the interactive drawing table control.

[0073] Here, the second feature extraction layer includes at least one downsampling layer to obtain a hierarchical feature map with a scale smaller than a preset initial scale, and ensure that the scales of at least one hierarchical feature map are different. Optionally, the downsampling layer is used to segment the initial feature map and splice the segmented feature maps in the channel dimension to reduce the scale of the initial feature map and increase the number of channels of the initial feature map.

[0074] Step 203: Input the initial feature map and at least one hierarchical feature map into a feature fusion layer in the image processing model to obtain a target user image output by the feature fusion layer.

[0075] Among them, the feature fusion layer is used to upsample the input feature maps of different scales into upsampled feature maps with a preset initial scale, and perform feature fusion on each upsampled feature map to obtain the target user image, that is, to fuse the feature maps of different hierarchical scales extracted by the first feature extraction layer and the second feature extraction layer.

[0076] In one embodiment, the initial feature map and at least one hierarchical feature map are input into an upsampling layer in a feature fusion layer to obtain a plurality of upsampling feature maps output by the upsampling layer; each upsampling feature map is input into a feature splicing layer in a feature fusion layer to obtain a target user image output by the feature splicing layer.

[0077] Considering that in the feature extraction process of the second feature extraction layer, the resolution of the hierarchical feature map is continuously reduced, which will cause the loss of some detail features; based on this, the input feature maps of different scales are upsampled to upsampled feature maps with a scale of a preset initial scale, and the upsampled feature maps are feature fused to obtain the target user image, that is, the feature maps of different scales are fused, and the target user image with deep semantic information and shallow feature information is generated while reducing the loss of detail features, thereby improving its expression ability, and then improving the recognition accuracy of the user posture; and because the input feature maps of different scales are upsampled to upsampled feature maps with a scale of a preset initial scale, the scale of the target user image is the preset initial scale, that is, a high-resolution target user image is generated, thereby improving its expression ability, and then improving the recognition accuracy of the user posture. That is, the feature fusion operation can reduce the loss of some features and improve the expression ability of the model.

[0078] In one embodiment, any hierarchical feature map is extracted based on the hierarchical feature extraction layer in the second feature extraction layer, and the hierarchical feature map is extracted based on the following method:

[0079] The initial feature map input to the hierarchical feature extraction layer is input to the downsampling layer in the hierarchical feature extraction layer to obtain the downsampling feature map output by the downsampling layer; the downsampling layer is used to segment the initial feature map and concatenate the segmented feature maps in the channel dimension to reduce the scale of the initial feature map and increase the number of channels of the initial feature map;

[0080] The downsampled feature map is input to the third feature extraction layer in the hierarchical feature extraction layer to obtain a hierarchical feature map output by the third feature extraction layer; the scale of the hierarchical feature map is the same as the scale of the downsampled feature map.

[0081] It should be noted that the second feature extraction layer includes at least one hierarchical feature extraction layer connected in sequence to extract at least one hierarchical feature map.

[0082] Here, the downsampling layer is used to perform downsampling processing to generate a hierarchical representation. In one embodiment, the downsampling layer is used to segment the initial feature map, and splice the segmented feature map in the channel dimension, so that the scale of the initial feature map is reduced to half of the original scale, and the number of channels of the initial feature map is increased to twice the original scale; that is, the scale of the downsampled feature map is half of the scale of the initial feature map, and the number of channels of the downsampled feature map is twice the number of channels of the initial feature map. Exemplarily, the downsampling layer is used to extract four small matrices from a matrix of the initial feature map at intervals, and then splice (concat) the four matrices in the channel dimension, and then after layer normalization (LayerNorm) processing, map them into 2 channels through a linear layer, based on which, the scale of the initial feature map is reduced to half of the original scale, and the number of channels of the initial feature map is increased to twice the original scale.

[0083] Here, the third feature extraction layer is used to extract features from the downsampled feature map to obtain a hierarchical feature map without changing the scale of the downsampled feature map. If the third feature extraction layer is followed by a hierarchical feature extraction layer, the hierarchical feature map is used as the initial feature map of the next hierarchical feature extraction layer.

[0084] like Figure 2 As shown, the second feature extraction layer includes three hierarchical feature extraction layers connected in sequence, namely a first-level feature extraction layer, a second-level feature extraction layer and a third-level feature extraction layer. Any hierarchical feature extraction layer includes a downsampling layer and a third feature extraction layer. Specifically, the initial feature map is input into the first-level feature extraction layer to obtain a first-level feature map output by the first-level feature extraction layer. The first-level feature map is input into the second-level feature extraction layer to obtain a second-level feature map output by the second-level feature extraction layer. The second-level feature map is input into the third-level feature extraction layer to obtain a third-level feature map output by the third-level feature extraction layer. Then, the initial feature map, the first-level feature map, the second-level feature map and the third-level feature map of different scales are input into the feature fusion layer to obtain the target user image output by the feature fusion layer.

[0085] The embodiment of the present invention inputs the initial feature map input to the hierarchical feature extraction layer into the downsampling layer in the hierarchical feature extraction layer to obtain a downsampling feature map output by the downsampling layer, and the downsampling layer is used to segment the initial feature map and splice the segmented feature map in the channel dimension to reduce the scale of the initial feature map and increase the number of channels of the initial feature map, thereby inputting the downsampling feature map into the third feature extraction layer in the hierarchical feature extraction layer to obtain a hierarchical feature map with stronger representation capability output by the third feature extraction layer, that is, to improve its expression capability, thereby improving the recognition accuracy of user posture.

[0086] In one embodiment, considering that the downsampling layer is used to segment the initial feature map, and the segmented feature maps are spliced ​​in the channel dimension to increase the number of channels of the initial feature map; based on this, the number of channels of the initial feature map is a preset initial channel number, the initial feature map and at least one hierarchical feature map are input into the feature fusion layer in the image processing model to obtain a target user image output by the feature fusion layer, including: inputting the initial feature map and at least one hierarchical feature map into the channel number unification layer in the feature fusion layer to obtain the initial feature map with unified channel numbers and at least one hierarchical feature map with unified channel numbers output by the channel number unification layer; inputting the initial feature map with unified channel numbers and at least one hierarchical feature map with unified channel numbers into the upsampling layer in the feature fusion layer to obtain multiple upsampled feature maps output by the upsampling layer; inputting each upsampled feature map into the feature splicing layer in the feature fusion layer to obtain the target user image output by the feature splicing layer.

[0087] Among them, the channel number unification layer is used to convert the input feature maps with different channel numbers into feature maps with a preset initial channel number. That is, the channel numbers of feature maps with different channel numbers are unified to the preset initial channel number. Based on this, it is convenient to fuse the feature maps with different channel numbers extracted by the first feature extraction layer and the second feature extraction layer. That is, the channel numbers of the initial feature map after the channel number is unified and the channel number of at least one level feature map after the channel number is unified are both the preset initial channel number.

[0088] In one embodiment, the layer with the same number of channels can be constructed by MLP (Multilayer Perceptron).

[0089] It should be noted that since the downsampling layer is used to segment the initial feature map and splice the segmented feature map in the channel dimension to increase the number of channels of the initial feature map, before feature fusion, it is necessary to unify the number of channels of the initial feature map and at least one level feature map to facilitate the subsequent better fusion of feature maps of different scales.

[0090] Here, the upsampling layer is used to upsample the initial feature map after the number of channels is unified and the at least one level feature map after the number of channels is unified into an upsampled feature map with a scale of a preset initial scale.

[0091] Here, the feature concatenation layer is used to concatenate the upsampled feature maps of the same scale and the same number of channels to obtain the target user image.

[0092] like Figure 3As shown, the initial feature map and the three-level feature maps are F1, F2, F3 and F4 respectively. The feature fusion layer includes a channel number unification layer, an upsampling layer and a feature concatenation layer in the dotted box, wherein the scale of the feature map processed by the upsampling layer is increased. Specifically, the initial feature map F1 and the three-level feature maps F2, F3, F4 are input into the feature fusion layer to obtain the target user image F output by the feature fusion layer.

[0093] In one embodiment, the preset initial scale may be one quarter of the input initial user image, so that the scale of the upsampled feature map obtained by upsampling may be one quarter of the initial user image.

[0094] In an embodiment of the present invention, an initial feature map and at least one hierarchical feature map are input into a channel number unification layer in a feature fusion layer to obtain an initial feature map with unified channel numbers and at least one hierarchical feature map with unified channel numbers output by the channel number unification layer, and the initial feature map with unified channel numbers and at least one hierarchical feature map with unified channel numbers are input into an upsampling layer in the feature fusion layer, so that before feature fusion is performed, the initial feature map and at least one hierarchical feature map are firstly unified in channel numbers to facilitate better fusion of feature maps of different scales, and finally obtain a target user image with stronger representation ability, that is, improve its expression ability, and further improve the recognition accuracy of user posture.

[0095] In one embodiment, the downsampled feature map is input into the third feature extraction layer in the hierarchical feature extraction layer to obtain the hierarchical feature map output by the third feature extraction layer, including: inputting the downsampled feature map into the first self-attention layer in the third feature extraction layer to obtain the first feature map output by the first self-attention layer; inputting the first feature map into the second self-attention layer in the third feature extraction layer to obtain the hierarchical feature map output by the second self-attention layer.

[0096] Among them, the first self-attention layer is used to divide the downsampled feature map into multiple windows for self-attention processing respectively, so as to better extract local features, that is, improve its expression ability, and then improve the recognition accuracy of user posture.

[0097] In one embodiment, the first self-attention layer includes a W-MSA (Window Multi-head Self-Attention) module, so that while the downsampled feature map is divided into multiple windows for self-attention processing, the computational complexity can be further reduced, thereby reducing the amount of calculation and improving the recognition efficiency of the user's posture. For example, the downsampled feature map is divided into multiple windows of 7*7 size for self-attention processing.

[0098] Optionally, the first self-attention layer includes a first normalization layer (LayerNorm), a W-MSA module, a first feature fusion module, a second normalization layer, a multi-layer perceptron (MLP) layer, and a second feature fusion module connected in sequence. Specifically, the downsampled feature map is input into the first normalization layer, and then the feature map output by the first normalization layer is input into the W-MSA module, and then the feature map output by the W-MSA module and the downsampled feature map are input into the first feature fusion module, the feature map output by the first feature fusion module is input into the second normalization layer, the feature map output by the second normalization layer is input into the multi-layer perceptron layer, the feature map output by the multi-layer perceptron layer and the feature map output by the first feature fusion module are input into the second feature fusion module, and the first feature map output by the second feature fusion module is obtained.

[0099] Among them, the second self-attention layer is used to divide the first feature map into multiple windows for self-attention processing, and to transfer information between the multiple windows after the first feature map is divided, so as to better perform global feature extraction, and can extract the contextual dependencies of different positions in the initial user image, that is, improve its expressive ability, and thus improve the recognition accuracy of user posture.

[0100] In one embodiment, the second self-attention layer includes a SW-MSA (Shifted Windows Multi-head Self-Attention) module, so that while the downsampled feature map is divided into multiple windows for self-attention processing respectively, information can be transferred between the multiple windows after the first feature map is divided, so that the contextual dependencies of different positions in the initial user image can be extracted.

[0101] Optionally, the second self-attention layer includes a third standardization layer, a SW-MSA module, a third feature fusion module, a fourth standardization layer, a multi-layer perceptron layer and a fourth feature fusion module connected in sequence. Specifically, the first feature map is input into the third standardization layer, and then the feature map output by the third standardization layer is input into the SW-MSA module, and then the feature map output by the SW-MSA module and the first feature map are input into the third feature fusion module, and the feature map output by the third feature fusion module is input into the fourth standardization layer, and the feature map output by the fourth standardization layer is input into the multi-layer perceptron layer, and the feature map output by the multi-layer perceptron layer and the feature map output by the third feature fusion module are input into the fourth feature fusion module to obtain the hierarchical feature map output by the fourth feature fusion module.

[0102] Optionally, the third feature extraction layer may include a plurality of consecutive first self-attention layers and second self-attention layers. For example, the third feature extraction layer includes a first self-attention layer, a second self-attention layer, a first self-attention layer, and a second self-attention layer connected in sequence.

[0103] In an embodiment of the present invention, the downsampled feature map is input to the first self-attention layer in the third feature extraction layer to obtain the first feature map output by the first self-attention layer, and the first self-attention layer is used to divide the downsampled feature map into multiple windows for self-attention processing respectively, and the first feature map is input to the second self-attention layer in the third feature extraction layer to obtain the hierarchical feature map output by the second self-attention layer, and the second self-attention layer is used to divide the first feature map into multiple windows for self-attention processing, and information is transferred between the multiple windows after the first feature map is divided, so that the local features and global features of the input initial user image can be effectively extracted, that is, its expression ability is improved, thereby improving the recognition accuracy of the user posture.

[0104] In one embodiment, an initial user image is input into a first feature extraction layer in an image processing model to obtain an initial feature map output by the first feature extraction layer, including: dividing the initial user image into multiple image blocks; superimposing the multiple image blocks in a channel dimension to obtain an original feature map; the scale of the original feature map is a preset initial scale; the original feature map is input into the first feature extraction layer to obtain an initial feature map output by the first feature extraction layer.

[0105] Specifically, the initial user image is divided into a plurality of image patches, thereby obtaining a plurality of image patches of a preset initial scale. Optionally, the initial user image is divided into a plurality of image patches of the same size. For example, the initial user image is divided into four image patches of the same size, so that for an initial user image of an input size of H×W×C, after division and superposition, an output size of For example, if the initial user image is an RGB image, multiple image blocks are superimposed on the color channel dimension to obtain the original feature map.

[0106] In one embodiment, the first feature extraction layer includes a linear embedding layer, so that the original feature map is feature mapped to obtain an initial feature map. Optionally, the first feature extraction layer includes a linear embedding layer, a first self-attention layer, and a second self-attention layer connected in sequence; specifically, the original feature map is input into the linear embedding layer, and the feature map output by the linear embedding layer is input into the first self-attention layer to obtain a second feature map output by the first self-attention layer; the first self-attention layer is used to divide the input feature map into multiple windows for self-attention processing respectively; the second feature map is input into the second self-attention layer to obtain the initial feature map output by the second self-attention layer; the second self-attention layer is used to divide the second feature map into multiple windows for self-attention processing, and to enable information transfer between the multiple windows after the second feature map is divided.

[0107] In one embodiment, the initial user image is segmented into a plurality of image blocks, and the linear embedding layer can be implemented by a two-dimensional convolution.

[0108] The embodiment of the present invention divides the initial user image into multiple image blocks, superimposes the multiple image blocks in the channel dimension to obtain an original feature map, and inputs the original feature map to the first feature extraction layer to obtain an initial feature map output by the first feature extraction layer, thereby more accurately capturing feature information at different positions in the initial user image, further improving the feature extraction capability, and further improving the image feature representation capability, and ultimately improving the accuracy of interactive drawing table control.

[0109] In one embodiment, the posture recognition model is trained based on a loss function, and the loss function includes at least one of a center point loss function, a center point offset loss function, and a target width and height loss function. The posture recognition model includes a posture recognition layer, and the posture recognition layer predicts the heat map, the center point, and the width and height respectively through three different convolution operations.

[0110] In one embodiment, the real target center points are processed by the Gaussian kernel function to generate a real heat map, the number of which is the number of target categories.

[0111] During training, the target center point prediction heat map output by the model is compared with the actual heat map. The center point loss function is as follows:

[0112]

[0113] Where, L k represents the center point loss value; N represents the number of key points in the image; Represents the predicted value on the heat map; It means that for category C, an object of category C is detected in the current (x, y) coordinate; α and β represent hyperparameters, for example, they are set to 2 and 4 respectively.

[0114] In a specific embodiment, after the target center point predicted on the sampled feature map is remapped to the original image, there will be a certain displacement deviation, so the displacement deviation of the predicted center point is introduced to compensate. The center point offset loss function of the predicted center point offset and the real center point offset is as follows:

[0115]

[0116] Where, L off represents the center point offset loss value; N represents the number of key points in the image; p represents the center point of the target box; O p ~ represents the predicted center point offset; R represents the downsampling multiple; Indicates the actual offset.

[0117] In one embodiment, assuming that the category is C k The bounding box of object k is Then the width and height of the target object can be expressed as: The target width and height loss function for calculating width and height is as follows:

[0118]

[0119] Where, L size Represents the target width and height loss value; N represents the number of key points in the image; Represents the predicted value of the target width and height; S k Indicates the true value of the target width and height.

[0120] In one embodiment, the loss function of the gesture recognition model is as follows:

[0121] L=L k +L off +L size .

[0122] Optionally, during the model training phase, you can set L off =1,L size =0.1.

[0123] The interactive drawing table provided by the present invention is described below. The interactive drawing table described below corresponds to the interactive drawing table control method based on human body posture recognition described above.

[0124] See also Figure 4 , Figure 4 The interactive drawing table provided in the embodiment of the present invention is a schematic diagram of an embodiment of the present invention. The interactive drawing table is equipped with a visual camera. The interactive drawing table includes:

[0125] The acquisition module 410 is used to acquire the initial user images of the user to be monitored at multiple consecutive moments based on the visual camera;

[0126] The image processing module 420 is used to input the initial user image into the image processing model to obtain the target user image of the user to be monitored output by the image processing model;

[0127] The posture recognition module 430 is used to input the target user image into the posture recognition model to obtain the user posture recognition result of the user to be monitored output by the posture recognition model; the posture recognition model is trained based on the sample user image and the user posture label corresponding to the sample user image;

[0128] An interactive display module 440 is used to display corresponding posture correction information on the interactive display interface of the interactive drawing table based on the user posture recognition result, so that the user to be monitored can perform posture correction according to the posture correction information;

[0129] Among them, the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

[0130] The interactive drawing table provided by the present invention processes the initial user image collected by the visual camera through the first feature extraction layer, the second feature extraction layer and the feature fusion layer in the image processing model, optimizes the image information in the initial user image, and further highlights the detail features in the target user image, and further performs posture recognition on the target user image through a pre-trained posture recognition model to obtain an accurate user posture recognition result, and further controls the interactive drawing table based on the accurate user posture recognition result. Therefore, the artificial intelligence of the image processing model and the posture recognition model can improve the recognition accuracy of the user posture, thereby improving the accuracy of the interactive drawing table control based on human posture recognition.

[0131] Optional, see Figure 5 , Figure 5 is a structural diagram of an electronic device provided by the present invention. An embodiment of the present invention provides an electronic device 500, including a memory 510, a processor 520, and a computer program 511 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 511, the following steps are implemented:

[0132] Acquire initial user images of the user to be monitored at multiple consecutive moments based on a visual camera;

[0133] Inputting the initial user image into the image processing model, and obtaining the target user image of the user to be monitored output by the image processing model;

[0134] The target user image is input into the posture recognition model to obtain the user posture recognition result of the monitored user output by the posture recognition model; the posture recognition model is trained based on the sample user image and the user posture label corresponding to the sample user image;

[0135] Based on the user posture recognition result, corresponding posture correction information is displayed on the interactive display interface of the interactive drawing table, so that the monitored user can perform posture correction according to the posture correction information;

[0136] Among them, the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An interactive drawing table control method based on human posture recognition, characterized in that: Applied to an interactive drawing table, the interactive drawing table is equipped with a visual camera, and the interactive drawing table control method based on human posture recognition includes: Acquire initial user images of the user to be monitored at multiple consecutive moments based on the visual camera; Inputting the initial user image into an image processing model to obtain a target user image of the user to be monitored output by the image processing model; Inputting the target user image into a gesture recognition model to obtain a user gesture recognition result of the user to be monitored output by the gesture recognition model; the gesture recognition model is trained based on sample user images and user gesture labels corresponding to the sample user images; Based on the user posture recognition result, corresponding posture correction information is displayed on the interactive display interface of the interactive drawing table, so that the user to be monitored performs posture correction according to the posture correction information; Among them, the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

2. The interactive drawing table control method based on human posture recognition according to claim 1, characterized in that: The step of inputting the initial user image into an image processing model to obtain a target user image of the user to be monitored output by the image processing model includes: Inputting the initial user image into a first feature extraction layer in the image processing model to obtain an initial feature map output by the first feature extraction layer; Inputting the initial feature map into the second feature extraction layer in the image processing model to obtain at least one hierarchical feature map output by the second feature extraction layer; the scale of the at least one hierarchical feature map is smaller than the preset initial scale, and the scales of the at least one hierarchical feature map are different; Inputting the initial feature map and the at least one hierarchical feature map into a feature fusion layer in the image processing model to obtain a target user image output by the feature fusion layer; The feature fusion layer is used to upsample the input feature maps of different scales into upsampled feature maps with a scale of the preset initial scale, and perform feature fusion on each of the upsampled feature maps to obtain the target user image.

3. The interactive drawing table control method based on human posture recognition according to claim 2, characterized in that: Any of the hierarchical feature maps is extracted based on the hierarchical feature extraction layer in the second feature extraction layer, and the hierarchical feature map is extracted based on the following method: Inputting the initial feature map input to the hierarchical feature extraction layer to the downsampling layer in the hierarchical feature extraction layer to obtain the downsampling feature map output by the downsampling layer; the downsampling layer is used to segment the initial feature map and splice the segmented feature maps in the channel dimension to reduce the scale of the initial feature map and increase the number of channels of the initial feature map; The downsampled feature map is input to a third feature extraction layer in the hierarchical feature extraction layer to obtain the hierarchical feature map output by the third feature extraction layer; the scale of the hierarchical feature map is the same as the scale of the downsampled feature map.

4. The interactive drawing table control method based on human posture recognition according to claim 3 is characterized in that: The number of channels of the initial feature map is a preset initial channel number; The step of inputting the initial feature map and the at least one hierarchical feature map into a feature fusion layer in the image processing model to obtain a target user image output by the feature fusion layer includes: Inputting the initial feature map and the at least one hierarchical feature map into the channel number unification layer in the feature fusion layer, obtaining the initial feature map with unified channel numbers and the at least one hierarchical feature map with unified channel numbers output by the channel number unification layer; the channel number unification layer is used to convert the input feature maps with different channel numbers into feature maps with the preset initial channel number; Inputting the initial feature map with a unified number of channels and at least one hierarchical feature map with a unified number of channels into the upsampling layer in the feature fusion layer to obtain a plurality of upsampling feature maps output by the upsampling layer; Each of the upsampled feature maps is input into the feature splicing layer in the feature fusion layer to obtain a target user image output by the feature splicing layer.

5. The interactive drawing table control method based on human posture recognition according to claim 3, characterized in that: The step of inputting the downsampled feature map to a third feature extraction layer in the hierarchical feature extraction layer to obtain the hierarchical feature map output by the third feature extraction layer comprises: Inputting the downsampled feature map into the first self-attention layer in the third feature extraction layer to obtain a first feature map output by the first self-attention layer; the first self-attention layer is used to divide the downsampled feature map into multiple windows for self-attention processing respectively; The first feature map is input into the second self-attention layer in the third feature extraction layer to obtain the hierarchical feature map output by the second self-attention layer; the second self-attention layer is used to divide the first feature map into multiple windows for self-attention processing, and to transfer information between the multiple windows after the first feature map is divided.

6. The interactive drawing table control method based on human posture recognition according to any one of claims 2 to 5, characterized in that: The step of inputting the initial user image into a first feature extraction layer in the image processing model to obtain an initial feature map output by the first feature extraction layer includes: Segmenting the initial user image into a plurality of image blocks; Superimposing the multiple image blocks in the channel dimension to obtain an original feature map; the scale of the original feature map is the preset initial scale; The original feature map is input into the first feature extraction layer to obtain an initial feature map output by the first feature extraction layer.

7. An interactive drawing table, characterized in that: The interactive drawing table is equipped with a visual camera, and the interactive drawing table comprises: An acquisition module, used for acquiring initial user images of the user to be monitored at multiple consecutive moments based on the visual camera; An image processing module, used for inputting the initial user image into an image processing model to obtain a target user image of the user to be monitored output by the image processing model; A gesture recognition module, used to input the target user image into a gesture recognition model to obtain a user gesture recognition result of the user to be monitored output by the gesture recognition model; the gesture recognition model is trained based on sample user images and user gesture labels corresponding to the sample user images; An interactive display module, configured to display corresponding posture correction information on an interactive display interface of the interactive drawing table based on the user posture recognition result, so that the user to be monitored can perform posture correction according to the posture correction information; Among them, the image processing model includes at least a first feature extraction layer, a second feature extraction layer and a feature fusion layer, the first feature extraction layer is used to extract an initial feature map of a preset initial scale, the second feature extraction layer is used to extract a hierarchical feature map, and the feature fusion layer is used to fuse the initial feature map and the hierarchical feature map.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the interactive drawing table control method based on human posture recognition as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the interactive drawing table control method based on human posture recognition as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the interactive drawing table control method based on human posture recognition as claimed in any one of claims 1 to 6 is implemented.

Images