Real-time multi-person body posture estimation method and system based on video stream transmission

Abstract

The invention discloses a real-time multi-person body posture estimation method and system based on video stream transmission, and the method comprises the steps: building a client side and a server side, building WebRTC communication connection between the server side and the client side, and enabling the client side to transmit the obtained video stream to the server side based on the WebRTC communication connection; the server caches the received video streams according to a time sequence, selects key frames according to the change of the optical streams, detects the human body from the cached key frames, estimates the postures of the detected human body frame by frame, and sends the estimated postures to the server; and drawing a human body posture estimation result on an original video frame for visual representation to form a complete skeleton structure, and returning the posture estimation result to the client through WebRTC communication connection. According to the invention, an efficient, universal and dynamically variable frame is provided for video stream transmission, efficient and low-delay attitude estimation result return can be realized, and the client can be ensured to check the processed video stream and attitude information in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and real-time communication, and in particular to a real-time multi-person human posture estimation method and system based on video stream transmission. Background Art

[0002] Human pose estimation is an important research direction in the field of computer vision. The purpose of human pose estimation is to detect and identify key points of the human body, such as the head, shoulders, wrists, knees, etc., from images or videos, and infer the human body's posture and movements based on these key points. Human pose estimation can be used in a variety of application scenarios such as human-computer interaction, motion analysis, virtual reality, and security monitoring.

[0003] However, the task of human posture estimation often consumes a lot of computing resources, and the processing of each frame of video (image) will bring a considerable time overhead. Although the parallel computing capabilities of the graphics computing unit (GPU) can basically meet the requirements of real-time processing, this relies on a large amount of hardware resources and cannot be deployed simply and conveniently. Real-time estimation and simple deployment are necessary in many scenarios. For example, in autonomous driving scenarios, it is necessary to estimate the motion state of pedestrians in real time to predict pedestrian paths and avoid collisions; in distance education, it is necessary to estimate students' postures in real time to analyze students' attention and evaluate teaching effectiveness; in video surveillance, it is necessary to estimate people's postures and behaviors in real time to evaluate potential dangers and issue warnings. In these scenarios, real-time human posture estimation in video streams is a key technology.

[0004] The purpose of real-time human pose estimation in video streams is to estimate the human pose from the video stream and to achieve efficient real-time transmission of the video streams before and after the pose estimation and the pose estimation results. The application scenarios of real-time human pose estimation in video streams are very wide, and can be applied to fields such as autonomous driving, distance education, and video surveillance. With the development of computer vision technology, the application scenarios of real-time human pose estimation in video streams are increasing, and the demand for real-time human pose estimation in video streams is also increasing.

[0005] However, real-time human pose estimation in video streams is a technology-intensive task, which requires the use of an efficient, universal and dynamically variable framework for video stream transmission. At the same time, estimating the human pose from the video stream requires real-time analysis and processing of the video stream. Real-time human pose estimation in video streams faces many challenges, such as high video stream latency, low quality, large illumination changes, severe occlusion, and complex multi-person interaction. In order to address these challenges, the present invention aims to build an efficient data stream processing framework, improve accuracy and robustness, reduce time and space complexity, and achieve real-time human pose estimation in video streams. Summary of the invention

[0006] In order to overcome the defects of the prior art, the present invention provides a real-time multi-person human posture estimation method and system based on video stream transmission, which can realize real-time and efficient estimation of high-precision human posture from video stream.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions.

[0008] On the one hand, the present invention provides a real-time multi-person human posture estimation method based on video stream transmission, comprising:

[0009] Building a client and a server, and establishing a WebRTC communication connection between the server and the client, based on which the client sends the obtained video stream to the server;

[0010] The server caches the received video stream in chronological order and selects key frames according to the changes in optical flow;

[0011] The server includes a human pose estimation model, which detects all human bodies from the cached key frames, and estimates the human pose including time series information for each detected human body frame by frame, and predicts the position information of key bone points of the human body;

[0012] The server draws the predicted position information of the skeleton key points on the original video frame for visualization, forming a complete skeleton structure;

[0013] The server transmits the posture estimation results back to the client through the WebRTC communication connection.

[0014] Furthermore, establishing a WebRTC communication connection between the server and the client includes:

[0015] Build a network server, which includes a web server and a signaling server;

[0016] The client accesses the web page address provided by the web server through a browser or a lightweight client with a browser kernel. The web server returns a web page containing WebRTC related functions. The client parses and executes the code corresponding to the web page to start establishing a WebRTC connection.

[0017] The client obtains local media and creates corresponding objects to associate local media data;

[0018] The client and server exchange information through the signaling server to determine the codec, format, and network transmission method of the media stream, thereby completing the WebRTC point-to-point connection between the client and the server.

[0019] Furthermore, the information exchanged between the client and the server through the signaling server includes SDP descriptions, candidate items, and network information, wherein the candidate items include the media types that the client and the server need to transmit respectively, the supported media stream codecs, formats, and / or network transmission protocols; the network information includes IP addresses and / or port numbers.

[0020] Furthermore, the WebRTC communication connection includes an established media channel and a data channel, wherein the media channel is used to transmit a media stream, and the data channel is used for data information.

[0021] Furthermore, the server caches the received video stream in chronological order and selects key frames according to the change of the optical flow, including:

[0022] A queue is preset in the server memory as a cache structure, which can cache up to N max Frame image;

[0023] Preset the base number of frames N and the minimum number of frames N for inputting the pose estimation model min , the number of video frames N needed to calculate the pose estimation based on optical flow new And cache it;

[0024] Create a cumulative optical flow counter FlowCounter and preset the optical flow threshold T f , the counter is used to accumulate the average optical flow between each frame and the previous frame. When the accumulated value of FlowCounter exceeds the preset optical flow threshold T f , mark the current frame as a key frame and clear FlowCounter.

[0025] Furthermore, the number of video frames N required for the pose estimation based on optical flow calculation is new include,

[0026] By N new =N+α(M(O)-T) to calculate the number of video frames needed; where T is the threshold of optical flow change, used to adjust the sensitivity to optical flow change; α is the proportional coefficient, used to adjust the amplitude of the frame number change;

[0027] By N adjusted =max(N min ,min(N max , N new )) constrains the number of video frames, where N adjusted is the final number of video frames, according to N adjusted Cache the current frame and delete the oldest cached frames older than this number;

[0028] N adjusted Recalculate N as the base frame number new =N adjusted，prev +α(M(O)-T), get the final number of video frames N new .

[0029] Furthermore, the server includes a human posture estimation model, which detects all human bodies from the cached key frames, and performs human posture estimation including time series information on each detected human body frame by frame, and predicts the position information of key skeletal points of the human body, including:

[0030] When a new video key frame is received and cached, human body detection is performed on the cached key frame using an existing human body object detection model to obtain the pixel coordinates and ranges of all human body objects in each frame image, and the results are cached;

[0031] The existing single-person pose estimation model is used to estimate the poses of all human objects in the key frames respectively. The horizontal and vertical directions of the image are divided into several intervals of equal width. The interval to which each key point belongs is determined according to the single-person pose estimation model to obtain the position information.

[0032] Furthermore, the server draws the predicted position information of the skeleton key points on the original video frame for visualization, forming a complete skeleton structure including:

[0033] Draw a marker at the position of each key point on the key frame through the image processing library;

[0034] According to the key point connection relationship, a straight line is drawn between the corresponding two key point marks to form a complete human skeleton;

[0035] The drawn key point markers and human skeleton layer are superimposed on the original video frame image to fuse the pose estimation result with the video frame.

[0036] Furthermore, the server transmits the posture estimation result back to the client via the WebRTC communication connection, including:

[0037] The posture estimation result is sent back to the client in the form of a data packet through the data channel in WebRTC;

[0038] The video frame image with the skeleton information superimposed is transmitted back to the client through the media channel in WebRTC.

[0039] In another aspect, the present invention provides a real-time multi-person human posture estimation system based on video streaming transmission, comprising:

[0040] The client is used to collect video streams, establish a WebRTC communication connection with the server, and visualize the human posture estimation results;

[0041] The server includes a cache module, a key frame selection module, a human body detection module, a human body posture estimation module and a visualization module; the cache module is used to cache the received video stream in time sequence; the key frame selection module is used to select key frames according to the change of optical flow; the human body detection module is used to detect all human bodies from the cached video frames; the human body posture estimation module is used to perform posture estimation including time series information on each detected human body and predict the position information of key bone points of the human body; the visualization module is used to draw the predicted bone key points on the video frame for visualization to form a complete skeleton structure;

[0042] The WebRTC communication connection module is used to establish a communication connection between the client and the server, including a media channel and a data channel. The media channel is used to transmit the video stream, and the data channel is used to transmit the data information of the posture estimation result.

[0043] Beneficial effects of the present invention:

[0044] The present invention is aimed at scenarios where there is a need for real-time human posture estimation but the hardware deployment is expensive. On the client side, only lightweight devices such as mobile phones and microcomputers are needed to complete real-time multi-person human posture estimation through video streaming, which reduces the deployment difficulty and cost, and can promote the implementation of human posture estimation applications.

[0045] The server mainly provides human posture estimation services. In addition, the server can also perform further analysis and visualization of posture estimation results to further assist specific applications. All data on the server is closed, reducing the risk of privacy leakage.

[0046] The present invention establishes a WebRTC communication connection between the client and the server, provides an efficient, universal and dynamically changeable framework for video stream transmission, and can achieve efficient and low-latency feedback of posture estimation results, ensuring that the client can view the processed video stream and posture information in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flow chart of a method for real-time multi-person human posture estimation based on video streaming provided by an embodiment of the present invention;

[0049] Figure 2 A flowchart of real-time transmission and posture estimation of video streams provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of a human body posture estimation process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to clearly illustrate the purpose, technical solutions and advantages of the present invention, the technical solutions of the present invention will be clearly and completely described through implementation methods with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Terminology explanation:

[0053] NAT (Network Address Translation) is a network function that allows multiple hosts to share one IP address. On the one hand, it solves the problem of IP address shortage, and on the other hand, it hides the real IP address of the intranet host and enhances network security. However, NAT also makes it difficult to obtain the real address of the peer in the process of establishing a peer connection, making it difficult to establish a connection.

[0054] STUN (Session Traversal Utilities for NAT) is a network protocol that allows clients behind NAT (or multiple NATs) to find out their public addresses, what type of NAT they are behind, and the Internet port that NAT binds to a local port. This information is used to establish UDP communication between two hosts behind NAT routers; the communication is end-to-end, and once the communication is established, STUN can no longer intervene.

[0055] The TURN (Traversal Using Relays around NAT) protocol is an extension of STUN. When the real address behind NAT cannot be obtained, it allows one or more hosts to use relay services to transmit messages to the other end, that is, a TURN server acts as a "middleman". In this way, the two parties to communicate establish their own connections with the TURN server located on the public Internet for communication, and the server is responsible for forwarding data between the two parties. Unlike STUN, since the TURN server acts as a middleman, it must continue to intervene to keep the communication going.

[0056] ICE (Interactive Connectivity Establishment) is a framework for establishing end-to-end connections. It integrates protocols such as STUN and TURN to help both ends of the network establish peer-to-peer connections.

[0057] WebRTC (Real-time communication for the web) is a collection of protocol specifications, standards, APIs, and concepts for establishing end-to-end real-time data communications. These peers are usually composed of two browsers, but WebRTC can also be used to establish a communication connection between a client and a server, or between any other devices that can implement the WebRTC standard.

[0058] Optical flow is a technique used to estimate the speed and direction of pixel movement in an image sequence. It infers the movement of an object by analyzing the brightness changes of pixels in adjacent frames. Its basic assumption is that the brightness of an object in an image remains relatively unchanged between frames as it moves. There are two common algorithms for optical flow: the Lucas-Kanade method is suitable for motion estimation in local areas, has fast calculation speed, and is suitable for real-time applications; the Horn-Schunck method estimates the smooth motion of the entire image through global optimization, which is suitable for complex scenes but has a high calculation cost.

[0059] Embodiment 1:

[0060] This embodiment provides a real-time multi-person human posture estimation method based on video stream transmission, comprising:

[0061] S1: Building a client and a server including a human posture estimation server, and establishing a WebRTC communication connection between the server and the client, based on which the client sends the obtained video stream to the server; a media channel and a data channel are established based on the WebRTC communication connection, the media channel is used to transmit the media stream, and the data channel is used to transmit data information;

[0062] S2: The server caches the received video stream in chronological order and selects key frames according to the changes in optical flow;

[0063] S3: The server detects all human bodies from the cached video frames and performs posture estimation including time series information for each detected human body. It calculates the position information of 17 key bone points of the human body through the existing posture estimation model, performs posture estimation frame by frame, and caches the posture estimation results.

[0064] S4: After the pose estimation is completed, the server draws the predicted skeleton key points on the video frame for visualization, which is displayed as clear circles and line segments to form a complete skeleton structure; the visualization result is superimposed on the original video frame to facilitate intuitive display of the pose estimation result;

[0065] S5: The server transmits the pose estimation results back to the client through the media channel and the data channel. The server transmits the key point coordinates, optical flow change values ​​and other metadata of the pose estimation back to the client in a custom format through the data channel in WebRTC. The server transmits the processed video stream with the skeleton information superimposed in real time through the media channel to ensure that the client can view the pose estimation results with low latency.

[0066] Specifically:

[0067] Establishing a WebRTC communication connection between the client and the server in S1 is a relatively complicated process. First, a network server is set up, including a Web page server and a signaling server. The client accesses the Web service through a browser to start establishing a WebRTC connection, obtains local media, and associates media with data streams. Subsequently, the client and server exchange conversation descriptions and network information through the signaling server, confirm parameters such as audio and video codecs, video stream resolution, frame rate, and network transmission protocols, and determine the client and server's respective addresses, ports, and other network information based on the ICE framework, thereby ultimately establishing a peer-to-peer connection between the client and the server and starting to transmit video streams. Specifically,

[0068] S101: Building a network server, including a web server and a signaling server;

[0069] Based on the specific application, a web page server is built and a front-end website using the WebRTC API connection service is constructed. The front-end network is the client in this embodiment, and the client website is deployed using a web service framework such as Node.js or Python Flask. It should be noted that the client generally includes a browser page or a lightweight client program such as a mobile phone or a microcomputer, which has the functions of collecting video streaming media, establishing connections, and visualizing the results.

[0070] In addition, the constructed signaling server is used to exchange various types of information between peers. The signaling server can be implemented based on protocols such as WebSocket and is responsible for coordinating the communication process. Its functions include exchanging description information and network information of the peers. It can be located on the same server host as the Web page server.

[0071] S102: The client accesses the Web page server;

[0072] The client accesses the web page address of the web server through the browser, and the web server sends a WebRTC connection request to the human posture estimation server. The web server returns a web page containing WebRTC related functions, and the client browser parses and displays the web page and executes the JavaScript code in it, thereby starting to establish a WebRTC connection.

[0073] S103: The client obtains local media and associates the media with data streams;

[0074] The client creates an RTCPeerConnection object, obtains local audio and video streams through the getUserMedia() method, and binds them to the RTCPeerConnection object for transmission in the WebRTC connection.

[0075] The client also establishes a data channel on the WebRTC connection through the createDataChannel() method to facilitate the transmission of subsequent posture estimation results.

[0076] S104: The client and the server exchange information through the signaling server to determine the codec, format, and network transmission method of the media stream;

[0077] The client and server will exchange SDP (Session Description Protocol) descriptions and candidate options through signaling services. SDP is a network protocol used to describe multimedia sessions. It defines the media type, media format, transmission protocol, IP address and port used in multimedia sessions. SDP itself does not transmit media data, but is used for session initialization and negotiation. The candidate options include the media type that each end needs to transmit, the supported media stream codec and format, network transmission protocol, etc. By comparing the candidate lists supported by both ends, the best options that can be supported by both ends are jointly agreed, including audio and video codecs, video stream resolution and frame rate, etc.

[0078] The audio codec includes but is not limited to AAC (Advanced Audio Coding), Opus, etc., and the video codec includes but is not limited to AVC (H.264), VP8 (Video Processor 8), etc. The codec of the above media stream can encode and compress the data before transmission, so that the bandwidth required for audio and video data of the same length and quality during transmission is smaller, and the original quality audio and video can be obtained by decoding after the transmission is completed. This encoding and decoding transmission process can ensure the high quality and low latency transmission of audio and video data, and at the same time, it also requires that the same type of codec can be used between the sending and receiving ends. The video stream resolution, frame rate, etc. can be dynamically adjusted according to network conditions and device performance to ensure the best user experience.

[0079] In addition, the client and the server will exchange ICE candidates, that is, their respective network information, including IP addresses, port numbers, etc., through the signaling server. This information is used to help the client and the server determine each other's specific location on the network in order to establish an end-to-end connection. The role of the signaling server in this step is to exchange information between the client and the server.

[0080] S105: Establishing a peer-to-peer connection between the client and the server;

[0081] Once the conversation description and network information are exchanged and negotiated through the signaling server, a WebRTC point-to-point connection will be established between the client and the server so that the client and the server can communicate directly and exchange media stream data, such as audio and video streams. In this embodiment, it is mainly used to transmit video streams. In addition, data in a custom format can also be transmitted through the data channel, such as posture estimation result data.

[0082] Furthermore, during the WebRTC connection process, the client and server will continuously exchange SDP descriptions and candidates. When the media status and network conditions change, the two can re-exchange information and negotiate the codec, format, and network transmission method of the media stream, etc., which enables the WebRTC connection to dynamically adjust and adapt to various network changes.

[0083] S2: The server caches the received video stream and selects key frames.

[0084] In this step, the server first caches the video stream received from the client, and then selects key frames to facilitate the subsequent steps of human posture estimation of the video frames. In the subsequent posture estimation, a series of cached images (i.e., continuous video frames) can be input, so that the model can have some timing information, because human movements are continuous, and the movements in the previous series of image frames are helpful for estimating the current frame, which can make the estimation result more accurate.

[0085] However, too much cache will lead to excessive memory overhead, and too large input will also increase the computational overhead of the human posture estimation model, thus affecting the real-time performance of the system. In order to ensure real-time performance, control the cache size, and ensure that the selected key frames can fully reflect the important action changes in the video stream, a dynamic selection algorithm based on optical flow is used to control the number of video frames used for posture estimation in the cache, while ensuring real-time performance and retaining as much timing information as possible. In addition, since the number of video frames is generally high, human movements may not necessarily change significantly in a small series of images; therefore, the calculated optical flow can also be used to select key frames whose changes exceed a preset threshold for posture estimation, rather than processing each frame, to reduce the inference overhead of the posture estimation model. The specific steps include,

[0086] S201: Building a cache queue;

[0087] First, a queue is used as a cache structure in the server memory. This cache structure can cache up to N max Frame image (N max Specified by the user according to the specific scenario). When the server receives a series of frame images in chronological order, it inserts frame images from the head of the queue in sequence. When the number of frame images is greater than N max When N, the frame images are deleted from the end in sequence, so that the total number is less than or equal to N. max , which controls the cache size.

[0088] S202: Select a certain number of video frames based on the optical flow and cache them;

[0089] First, the user determines a basic frame number N and a minimum frame number N for the input pose estimation model based on the specific usage scenario. min Whenever a new frame is obtained from the video stream, the sparse optical flow calculation method can be used to calculate the optical flow between the frame and the previous frame. The formula is expressed as O(F, F prev ), where F is the current frame, F prev For the previous frame, all optical flow vectors can be averaged to calculate the average magnitude M(O) of the optical flow.

[0090] The number of video frames required for pose estimation is calculated using the following formula 1:

[0091] N new =N+α(M(O)-T) Formula 1

[0092] Where T is the threshold of optical flow change, which is used to adjust the sensitivity to optical flow change and is specified by the user; α is the proportional coefficient, which is used to adjust the amplitude of the frame number change and is specified by the user; N new is the number of video frames calculated.

[0093] In addition, in order to prevent the number of frames from exceeding the cache range, the following formula 2 is used for constraint:

[0094] N adjusted =max(N min ,min(N max , N new )) Formula 2

[0095] Where N adjusted The final number of video frames is determined, and the current frame is cached according to this number and the earliest cached frames exceeding this number are deleted. adjusted All need to be calculated and may change. The N adjusted It will be used as the basic frame number for this calculation, that is, N in formula 1 for further calculation. The formula is as follows:

[0096] N new =N adjusted，prev +α(M(O)-T) Formula 3

[0097] S203: Select key frames based on optical flow;

[0098] Create a cumulative optical flow counter FlowCounter with an initial value of 0. This counter is used to accumulate the average optical flow between each frame and the previous frame. For each newly received frame F, the formula O(F, F prev ) Calculate the optical flow between the current frame and the previous frame, and obtain the average size of the optical flow M(O) (consistent with S202, no need to repeat the calculation). Add M(O) to FlowCounter, that is:

[0099] FlowCounter=FlowCounter+M(O) Formula 4

[0100] The user sets an optical flow threshold T according to the usage scenario f When the accumulated value of FlowCounter exceeds the threshold, the current frame is marked as a key frame and FlowCounter is cleared to zero. This ensures that a frame is selected as a key frame only when the motion between frames exceeds the threshold and has a significant change.

[0101] When a new frame enters the cache, the optical flow is recalculated and the counter is updated. Whenever the FlowCounter reaches or exceeds the threshold, the frame is selected as a key frame and the counter is reset. This dynamic selection of key frames can balance the frequency of frame selection and motion change detection.

[0102] See also Figure 3 In step S3, the server performs global human body detection from top to bottom in the cached video frames through the human object detection model, detects all human bodies, and performs separate posture estimation and post-processing for each detected human body using the human posture estimation model, that is, human posture estimation including timing information is performed for each detected human body. It should be noted that the human posture estimation model is a prior art, and there are many publicly available posture estimation models. The corresponding posture estimation model can be used for posture estimation according to specific needs. For example, if real-time performance is required, use the lightweight model PoseNet or AlphaPose; if high precision is required, the HRNet model or OpenPose model can be selected. Specifically including,

[0103] S301: Perform human body detection on the cached video frames using the existing human body object detection model. When a new video frame is received and cached, use the target detection API or other object detection tools provided by the MMDetection toolkit to detect human objects in the video frame, obtain the pixel coordinates and ranges of all human objects in each frame, and cache the results. Note that the detection tool needs to have the ability to track objects, that is, it can associate human objects belonging to the same character in a series of cached video frame sequences (generally identified by the same ID).

[0104] S302: Perform posture estimation on all human objects in the new video frame. Use the existing single-person posture estimation model to perform posture estimation on all human objects in the key frame. Today's more efficient single-person posture estimation model regards posture estimation as a classification task to predict key points on the x-axis and y-axis in the image coordinate system. Specifically, the horizontal and vertical directions of the image are divided into several intervals of equal width (the number of intervals is determined according to the image pixels and the required accuracy, generally between 64 and 256), and then the trained end-to-end single-person posture estimation model is used to determine which interval each key point belongs to. In this way, the continuous coordinate values ​​can be discretized into integer labels, thereby simplifying the problem of posture estimation and obtaining the position information of the key points of the human skeleton.

[0105] The single-person pose estimation model will predict the following 17 human skeleton key points: nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle. The prediction results of the above key points will be used for subsequent pose estimation and visualization.

[0106] In step S4, based on the 17 human skeleton key point information output by the human posture estimation model, the above human skeleton key points and skeleton connections are visualized on the key frame image using an image processing library (such as OpenCV, Matplotlib, etc.):

[0107] S401: Draw a mark at the position of each key point of the human skeleton. In this embodiment, the mark is a circle, and the color and size can be adjusted as needed to make it clearly visible on the image, thereby realizing visualization of the key points of the skeleton;

[0108] S402: according to the connection relationship of the key points (such as the left shoulder connected to the left elbow, the left elbow connected to the left wrist, etc.), a straight line is drawn between two corresponding key points to form a complete human skeleton;

[0109] S403: superimposing the drawn human skeleton key points and human skeleton layer onto the original key frame image to fuse the posture estimation result with the video frame, that is, visualizing the human skeleton key points, so that the user can see the superimposed posture estimation results in the video stream and intuitively understand the posture of the human body in each frame of the image.

[0110] S5. The server uses the media channel and data channel to transmit the posture estimation results back to the client. Through the WebRTC connection, the server can return the processed video frames and posture estimation results to the client to achieve real-time feedback.

[0111] S501: Posture estimation result is transmitted back. The posture estimation result is sent back to the client in the form of a data packet using the Data Channel in WebRTC. The Data Channel can transmit data in a custom format, such as the coordinates of skeleton key points, and return the x and y coordinates of each key point to the client as JSON or other format data for further processing by the client; other metadata is also returned to the client, including optical flow change values, key frame tags, frame processing time and other information, so that the client can understand the relevant information of the posture estimation process.

[0112] S502: Video stream transmission. The image with the skeleton information superimposed is transmitted back to the client through the media channel, and the client can directly display the processed video stream. The front-end page of the client uses Canvas or other image processing technology to display the video frame with the pose estimation result.

[0113] The above method of using the data channel and the media channel for data feedback at the same time can achieve efficient and low-latency feedback of posture estimation results, ensuring that the client can view the processed video stream and posture information in real time and can make corresponding responses or processing.

[0114] For example, in a classroom teaching scenario, surveillance videos of multiple classrooms can be streamed to a human posture estimation server, which estimates the postures of students in the classroom and analyzes the students' attention status based on the estimation results, and then feeds the results back to the teacher in the corresponding classroom. The benefits of this example are: no additional new equipment needs to be installed in existing classrooms, and the existing monitoring system can be used; there is no need to configure an independent system for each classroom, and the entire school or even multiple schools can share a set of back-end posture estimation services; all data are processed in a closed manner in the system, reducing the risk of privacy leakage.

[0115] Embodiment 2:

[0116] This embodiment provides a real-time multi-person human posture estimation system based on video stream transmission, comprising:

[0117] The client is used to collect video streams, establish a WebRTC communication connection with the server, and visualize the human posture estimation results;

[0118] The server includes a cache module, a key frame selection module, a human body detection module, a human body posture estimation module and a visualization module; the cache module is used to cache the received video stream in time sequence; the key frame selection module is used to select key frames according to the change of optical flow; the human body detection module is used to detect all human bodies from the cached video frames; the human body posture estimation module is used to perform posture estimation including time series information on each detected human body and predict the position information of key bone points of the human body; the visualization module is used to draw the predicted bone key points on the video frame for visualization to form a complete skeleton structure;

[0119] The WebRTC communication connection module is used to establish a communication connection between the client and the server, including a media channel and a data channel. The media channel is used to transmit the video stream, and the data channel is used to transmit the data information of the posture estimation result.

[0120] The above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A real-time multi-person human pose estimation method based on video streaming, include, Building a client and a server, and establishing a WebRTC communication connection between the server and the client, based on which the client sends the obtained video stream to the server; The server caches the received video stream in chronological order and selects key frames according to the changes in optical flow; The server includes a human pose estimation model, which detects all human bodies from the cached key frames, and estimates the human pose including time series information for each detected human body frame by frame, and predicts the position information of key bone points of the human body; The server draws the predicted position information of the skeleton key points on the original video frame for visualization, forming a complete skeleton structure; The server transmits the posture estimation results back to the client through the WebRTC communication connection.

2. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 1 is characterized in that: The establishing of a WebRTC communication connection between the server and the client includes: Build a network server, which includes a web server and a signaling server; The client accesses the web page address provided by the web server through a browser or a lightweight client with a browser kernel. The web server returns a web page containing WebRTC related functions. The client parses and executes the code corresponding to the web page to start establishing a WebRTC connection. The client obtains local media and creates corresponding objects to associate local media data; The client and server exchange information through the signaling server to determine the codec, format, and network transmission method of the media stream, thereby completing the WebRTC point-to-point connection between the client and the server.

3. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 2 is characterized in that: The information exchanged between the client and the server through the signaling server includes SDP description, candidate items and network information. The candidate items include the media type, supported media stream codecs, formats and / or network transmission protocols that the client and the server need to transmit respectively; the network information includes IP addresses and / or port numbers.

4. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 1 is characterized in that: The WebRTC communication connection includes an established media channel and a data channel, wherein the media channel is used to transmit a media stream, and the data channel is used for data information.

5. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 1 is characterized in that: The server caches the received video stream in time order and selects key frames according to the change of optical flow, including: A queue is preset in the server memory as a cache structure, which can cache up to N max Frame image; Preset the base number of frames N and the minimum number of frames N for inputting the pose estimation model min , the number of video frames N needed to calculate the pose estimation based on optical flow new And cache it; Create a cumulative optical flow counter FlowCounter and preset the optical flow threshold T f , the counter is used to accumulate the average optical flow between each frame and the previous frame. When the accumulated value of FlowCounter exceeds the preset optical flow threshold T f , mark the current frame as a key frame and clear FlowCounter.

6. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 5 is characterized in that: The number of video frames N required for pose estimation based on optical flow calculation new include, By N new =N+α(M(O)-T) to calculate the number of video frames needed; where T is the threshold of optical flow change, used to adjust the sensitivity to optical flow change; α is the proportional coefficient, used to adjust the amplitude of the frame number change; By N adjusted =max(N min ,min(N max ,N new )) constrains the number of video frames, where N adjusted is the final number of video frames, according to N adjusted Cache the current frame and delete the oldest cached frames older than this number; N adjusted Recalculate N as the base frame number new =N adjusted,prev +α(M(O)-T), get the final number of video frames N new .

7. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 1 is characterized in that: The server includes a human posture estimation model, which detects all human bodies from the cached key frames, and estimates the human posture containing time series information for each detected human body frame by frame, and predicts the position information of key bone points of the human body, including: When a new video key frame is received and cached, human body detection is performed on the cached key frame using an existing human body object detection model to obtain the pixel coordinates and ranges of all human body objects in each frame image, and the results are cached; The existing single-person pose estimation model is used to estimate the poses of all human objects in the key frames respectively. The horizontal and vertical directions of the image are divided into several intervals of equal width. The interval to which each key point belongs is determined according to the single-person pose estimation model to obtain the position information.

8. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 1 is characterized in that: The server draws the predicted position information of the skeleton key points on the original video frame for visualization, forming a complete skeleton structure including: Draw a marker at the position of each key point on the key frame through the image processing library; According to the key point connection relationship, a straight line is drawn between the corresponding two key point marks to form a complete human skeleton; The drawn key point markers and human skeleton layer are superimposed on the original video frame image to fuse the pose estimation result with the video frame.

9. The real-time multi-person human posture estimation method based on video streaming transmission according to claim 4 is characterized in that: The server transmits the posture estimation result back to the client via the WebRTC communication connection, including: The posture estimation result is sent back to the client in the form of data packets through the data channel in WebRTC; The video frame image with the skeleton information superimposed is transmitted back to the client through the media channel in WebRTC.

10. The real-time multi-person human posture estimation system based on video streaming transmission according to claim 1, characterized in that: include, The client is used to collect video streams, establish a WebRTC communication connection with the server, and visualize the human posture estimation results; The server includes a cache module, a key frame selection module, a human body detection module, a human body posture estimation module and a visualization module; the cache module is used to cache the received video stream in chronological order; The key frame selection module is used to select key frames according to the change of optical flow; the human body detection module is used to detect all human bodies from the cached video frames; The human body posture estimation module is used to perform posture estimation including time series information on each detected human body, and predict the position information of key bone points of the human body; the visualization module is used to draw the predicted bone key points on the video frame for visualization, so as to form a complete skeleton structure; The WebRTC communication connection module is used to establish a communication connection between the client and the server, including a media channel and a data channel. The media channel is used to transmit the video stream, and the data channel is used to transmit the data information of the posture estimation result.

Images