Human body key point identification method and system based on posture video data fusion

Abstract

The invention relates to the technical field of image processing, in particular to a human body key point recognition method and system based on posture video data fusion, and the method comprises the steps: receiving a key point recognition instruction, starting a data collection unit according to the key point recognition instruction, collecting posture video data through a main collection unit and an auxiliary collection unit, and extracting a first-frame posture image; detecting to obtain a preliminary detection frame, generating a posture video heat map, adjusting the preliminary detection frame by using the posture video heat map to obtain an adjusted detection frame, acquiring a detection main image and a detection auxiliary image from the posture video data according to the adjusted detection frame, and inputting the detection main image and the detection auxiliary image into a key point network; and obtaining a main prediction key point set and an auxiliary prediction key point set, performing identification based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and completing human body key point identification according to the identification key point set. The accuracy of human body key point recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and system for recognizing key points of a human body based on posture video data fusion. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, human key point recognition has become a core technology in many application fields such as video understanding, motion analysis, augmented reality, intelligent security, and sports rehabilitation. By accurately positioning key points such as human joints and limbs, it can achieve a fine-grained understanding of human posture, providing intuitive and reliable data support for behavior recognition, motion assessment, and human-computer interaction.

[0003] Traditional methods for identifying keypoints in the human body often rely on single-camera acquisition, resulting in limited field of view, occlusion sensitivity, low recognition accuracy, and poor real-time performance. Keypoint detection often suffers from inaccuracy in the presence of partial occlusion or complex backgrounds from a single viewpoint. Furthermore, traditional methods lack multimodal, multi-viewpoint synchronous triggering and coordination mechanisms, making it difficult to obtain complete and continuous pose information in large-scale scenes or dynamic motion. Therefore, improving the accuracy of human keypoint recognition is a critical issue that needs to be addressed. Summary of the Invention

[0004] The present invention provides a human body key point recognition method and system based on posture video data fusion, the main purpose of which is to improve the accuracy of human body key point recognition.

[0005] To achieve the above-mentioned object, the present invention provides a method for identifying key points of a human body based on posture video data fusion, comprising: Receive a key point recognition instruction, and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; Utilizing the main acquisition unit and the auxiliary acquisition unit to collect data of a preset target human body to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; Generating a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; Obtaining a detection main image and a detection auxiliary image from the posture video data according to the adjusted detection frame, and inputting the obtained detection main image and detection auxiliary image into a pre-built key point network to obtain a main prediction key point set and an auxiliary prediction key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

[0006] Optionally, the using the main acquisition unit and the auxiliary acquisition unit to collect data on a preset target human body to obtain posture video data includes: Based on a preset acquisition frequency, a preset acquisition time and the main acquisition unit, an image of the target human body is acquired to obtain a main acquisition image and generate a main acquisition signal, wherein a main acquisition signal is generated each time an image is acquired; Using the hardware trigger line to send the main acquisition signal to the auxiliary acquisition unit, and using the auxiliary acquisition unit to acquire an image of the target human body to obtain an auxiliary acquisition image, collecting the main acquisition images to obtain a main acquisition image set, and collecting the auxiliary acquisition images to obtain an auxiliary acquisition image set, wherein the main acquisition images in the main acquisition image set correspond to the auxiliary acquisition images in the auxiliary acquisition image set in a one-to-one manner; Acquire a main detection image from the main acquisition image set, and construct a pixel intensity histogram based on the acquired main detection image; The total number of pixels and the number of pixel points are obtained according to the pixel intensity histogram, and the average brightness of the pixels is calculated using the total number of pixels and the number of pixel points. The calculation formula of the average brightness of the pixels is as follows: in, Refers to the average brightness of pixels, Refers to the preset grayscale parameters, The grayscale parameter is The number of pixels when Refers to the total number of pixels; The posture video data is obtained according to the average brightness of the pixels, the main detection image and the auxiliary acquisition image set.

[0007] Optionally, the acquiring of posture video data according to the average pixel brightness, the primary detection image, and the auxiliary acquisition image set includes: Compare the average pixel brightness with a preset brightness threshold; If it is determined that the average pixel brightness is less than the brightness threshold, a corresponding auxiliary adjustment image is obtained from the auxiliary acquisition image set, and the corresponding auxiliary adjustment image is pseudo-color mapped to obtain a mapping acquisition image, wherein the corresponding auxiliary adjustment image is an auxiliary acquisition image corresponding to the main detection image; Converting the mapping acquisition image to obtain a brightness channel image, and calculating the mapping brightness value according to the brightness channel image; The original pixel value is obtained according to the main detection image, and the weighted adjustment brightness value is calculated using the original pixel value and the mapped brightness value, wherein the calculation formula of the weighted adjustment brightness value is as follows: in, Refers to the image horizontal axis parameter , the image vertical axis parameters are The weighted adjustment brightness value is Refers to the preset weighting parameters, Refers to the preset image horizontal axis parameters, Refers to the preset image vertical axis parameters, Refers to the image horizontal axis parameter , the image vertical axis parameters are The mapped brightness value when Refers to the image horizontal axis parameter , the image vertical axis parameters are The original pixel value at time ; Performing brightness replacement on the main detection image using the weighted adjusted brightness value to obtain a target main image; The posture video data is acquired according to the main acquisition image set, the auxiliary acquisition image set and the target main image.

[0008] Optionally, the detecting based on the extracted first-frame posture image to obtain a preliminary detection frame includes: Inputting the first-frame pose image into a pre-built detection model, and using the detection model to detect the input first-frame pose image to obtain a candidate frame set and a candidate score set, wherein the candidate frames in the candidate frame set correspond one-to-one to the candidate scores in the candidate score set; Extracting candidate scores from the candidate score set in sequence, and comparing the candidate scores with a preset score threshold; If the candidate score is less than the score threshold, the extracted candidate score is removed from the candidate score set to obtain an updated score set, the updated score set is used to update the candidate score set, and the updated candidate score set is used to return to the above step of sequentially extracting candidate scores from the candidate score set until the candidate score set is empty; If the candidate score is not less than the score threshold, the extracted candidate score is input into the pre-constructed target score set, and the extracted candidate score is removed from the candidate score set to obtain an updated score set, the candidate score set is updated using the updated score set, and the updated candidate score set is used to return to the above step of sequentially extracting candidate scores from the candidate score set until the candidate score set is an empty set, wherein the target score set is initially an empty set; A target candidate frame set is matched according to the target score set, and a preliminary detection frame is obtained according to the target candidate frame set.

[0009] Optionally, obtaining a preliminary detection frame according to the target candidate frame set includes: Extract target candidate frames from the target candidate frame set in sequence, calculate areas of the extracted target candidate frames to obtain candidate frame areas, and aggregate the candidate frame areas to obtain a candidate frame area set, wherein the areas of the candidate frames in the candidate frame area set correspond one-to-one to the target candidate frames in the target candidate frame set; A maximum candidate frame area is screened from the candidate frame area set, a maximum candidate frame is matched according to the screened maximum candidate frame area, and the maximum candidate frame is confirmed as a preliminary detection frame.

[0010] Optionally, generating a heat map based on the posture video data to obtain a posture video heat map includes: performing grayscale normalization on the posture video data to obtain a normalized image sequence; Extracting a first image and a second image in sequence from the normalized image sequence, removing the first image from the normalized image sequence to obtain a removed image sequence, performing calculations based on the extracted first image and second image to obtain a pixel-level absolute difference, and obtaining a pixel difference map based on the pixel-level absolute difference, wherein the first image is the first image in the normalized image sequence, and the second image is an image adjacent to the first image in the normalized image sequence; Using the eliminated image sequence to update the normalized image sequence, and returning to the above steps of sequentially extracting the first image and the second image from the normalized image sequence according to the updated normalized image sequence, until the number of images in the normalized image sequence is 1; A posture video heat map is obtained according to the pixel difference map.

[0011] Optionally, the performing calculation based on the extracted first image and second image to obtain a pixel-level absolute difference includes: Obtain a normalized first grayscale value of the first image, obtain a normalized second grayscale value of the second image, and calculate a pixel-level absolute difference based on the normalized first grayscale value and the normalized second grayscale value, wherein the calculation formula of the pixel-level absolute difference is as follows: in, Refers to the image horizontal axis parameter , the image vertical axis parameters are The absolute difference at the pixel level is The horizontal axis parameter of the first image is , the image vertical axis parameters are The normalized first grayscale value at The horizontal axis parameter of the second image is , the image vertical axis parameters are The normalized second grayscale value at .

[0012] Optionally, the adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame includes: Obtaining a preliminary horizontal coordinate, a preliminary vertical coordinate, a preliminary frame width, and a preliminary frame height according to the preliminary detection frame; Constructing a posture motion binary map according to the posture video heat map, and obtaining a motion pixel point set according to the preliminary horizontal coordinate, preliminary vertical coordinate, preliminary frame width, preliminary frame height and the posture motion binary map; Obtaining the upper left corner point and the lower right corner point from the moving pixel point set; The preliminary detection frame is adaptively adjusted according to the upper left corner point and the lower right corner point, the preliminary horizontal coordinate, the preliminary vertical coordinate, the preliminary frame width, and the preliminary frame height to obtain an adjusted detection frame.

[0013] Optionally, adaptively adjusting the preliminary detection frame according to the upper left corner point and the lower right corner point, the preliminary horizontal coordinate, the preliminary vertical coordinate, the preliminary frame width, and the preliminary frame height to obtain the adjusted detection frame includes: The upper left corner point, the lower right corner point, the preliminary horizontal coordinate and the preliminary vertical coordinate are globally mapped to obtain a global upper left corner point and a global lower right corner point, wherein the global upper left corner point is as follows: in, Refers to the global upper left corner, Refers to the initial horizontal axis, Refers to the horizontal coordinate of the upper left corner point, Refers to the initial vertical coordinate, Refers to the vertical coordinate of the upper left corner; The global lower right corner point is as follows: in, Refers to the global lower right corner, Refers to the horizontal coordinate of the lower right corner point, Refers to the vertical coordinate of the lower right corner; The adjustment frame width and the adjustment frame height are obtained according to the global upper left corner point and the global lower right corner point, and the adjustment frame width and the adjustment frame height are used to replace the preliminary frame width and the preliminary frame height of the preliminary detection frame to obtain the adjusted detection frame.

[0014] To achieve the above object, the present invention further provides a human key point recognition system based on posture video data fusion, comprising: a unit startup module, configured to receive a key point recognition instruction and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; A preliminary detection module is used to collect data of a preset target human body using the main acquisition unit and the auxiliary acquisition unit to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; A frame adjustment module is used to generate a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; a key recognition module, configured to obtain a detection primary image and a detection secondary image from the posture video data according to the adjusted detection frame, and input the obtained detection primary image and detection secondary image into a pre-built key point network to obtain a primary predicted key point set and a secondary predicted key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

[0015] In order to solve the above problem, the present invention further provides an electronic device, comprising: a memory storing at least one instruction; The processor executes the instructions stored in the memory to implement the above-mentioned human body key point recognition method based on posture video data fusion.

[0016] In order to solve the above problems, the present invention also provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is executed by a processor in an electronic device to implement the above-mentioned human key point recognition method based on posture video data fusion.

[0017] The present invention solves the problems described in the background technology. First, a key point recognition instruction is received, and a data acquisition unit is started according to the received key point recognition instruction. The data acquisition unit integrates a main acquisition unit and an auxiliary acquisition unit. The main acquisition unit and the auxiliary acquisition unit are connected by a hardware trigger line to ensure that the main acquisition unit and the auxiliary acquisition unit are almost simultaneously acquiring data, providing image data at the same time for subsequent operations. The overlapping area between the main acquisition unit and the auxiliary acquisition unit is 60°. The 60° overlapping area ensures that the target human body is always in the common visible range of the main acquisition unit and the auxiliary acquisition unit, reducing the omission of key points caused by the blind spot of a single perspective, and providing a reliable data basis for the subsequent acquisition of posture video data; secondly, the main acquisition unit and the auxiliary acquisition unit are used to acquire data on the target human body to obtain posture video data, and the main acquisition unit is acquired according to the image acquired by the auxiliary acquisition unit. The image of the set is adjusted to obtain posture video data, which improves the integrity and clarity of the target human body in the posture video data; then, the first frame posture image is extracted from the posture video data, and a preliminary detection frame is generated based on the first frame posture image. The preliminary detection frame is used to confirm the approximate position of the target human body, and the foundation is laid for the subsequent adjustment of the preliminary detection frame; further, the preliminary detection frame is adaptively adjusted to obtain an adjusted detection frame, and the preliminary detection frame is adjusted according to the posture video heat map to obtain the optimal detection frame representing the target human body, thereby improving the efficiency and accuracy of key point detection; finally, the detection main image and the detection auxiliary image are obtained according to the adjusted detection frame, and the obtained detection main image and detection auxiliary image are input into the key point network to obtain the main prediction key point set and the auxiliary prediction key point set, and the main prediction key point set and the auxiliary prediction key point set are compared to finally obtain the recognition key point set. Therefore, the present invention can improve the accuracy of human key point recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic flow chart of a method for identifying key points of a human body based on posture video data fusion provided by one embodiment of the present invention; Figure 2 A functional module diagram of a human body key point recognition system based on posture video data fusion provided by one embodiment of the present invention; Figure 3 A schematic structural diagram of an electronic device for implementing the method for identifying key points of a human body based on posture video data fusion provided by an embodiment of the present invention.

[0019] Description of reference numerals: 1. Electronic device; 10. Processor; 11. Memory; 12. Bus.

[0020] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0022] The embodiment of the present application provides a method for identifying key points of a human body based on the fusion of posture video data. The execution subject of the method for identifying key points of a human body based on the fusion of posture video data includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the method for identifying key points of a human body based on the fusion of posture video data can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0023] Reference Figure 1 FIG. 1 is a flow chart of a method for identifying key points of a human body based on posture video data fusion according to an embodiment of the present invention. In this embodiment, the method for identifying key points of a human body based on posture video data fusion includes: S1. Receive a key point recognition instruction, and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected by a hardware trigger line, and the overlapping area of the main acquisition unit and the auxiliary acquisition unit is 60°.

[0024] It can be explained that the key point recognition instruction refers to an instruction issued manually to start the data acquisition unit. For example, Xiao Zhang is an employee of a company. Now Xiao Zhang needs to perform key point recognition of the human body, so Xiao Zhang issues a key point recognition instruction and starts the data acquisition unit according to the key point recognition instruction. The data acquisition unit refers to a unit used to collect data from the target human body. The target human body refers to the target that needs to perform key point recognition of the human body. The data acquisition unit consists of a main acquisition unit and an auxiliary acquisition unit. The main acquisition unit refers to a camera used to collect data from the target human body. The main acquisition unit outputs an RGB image. The auxiliary acquisition unit refers to an auxiliary camera used to collect data from the target human body. The auxiliary acquisition unit outputs a thermal image. The thermal image refers to an image that maps the temperature of each position with the pixel grayscale value. The higher the pixel grayscale value in the thermal image, the higher the temperature of the corresponding position. The hardware trigger line is the cable used to connect the main acquisition unit and the auxiliary acquisition unit. When the main acquisition unit acquires an image, it generates a signal, which is transmitted to the auxiliary acquisition unit via the hardware trigger line. After receiving the signal, the auxiliary acquisition unit acquires the image. The hardware trigger line transmits signals with extremely low latency, typically less than 1 microsecond. This extremely low latency ensures that the main and auxiliary acquisition units acquire data at the same time. The overlapping area refers to the shared portion of the visible images of the main and auxiliary acquisition units. The respective visible images refer to the shooting ranges of the main and auxiliary acquisition units, and the shared portion refers to the overlap between the visible images of the main and auxiliary acquisition units.

[0025] S2. Utilize the main acquisition unit and the auxiliary acquisition unit to collect data on a preset target human body to obtain posture video data.

[0026] In detail, the main acquisition unit and the auxiliary acquisition unit are used to collect data of a preset target human body to obtain posture video data, including: Based on a preset acquisition frequency, a preset acquisition time and the main acquisition unit, an image of the target human body is acquired to obtain a main acquisition image and generate a main acquisition signal, wherein a main acquisition signal is generated each time an image is acquired; Using the hardware trigger line to send the main acquisition signal to the auxiliary acquisition unit, and using the auxiliary acquisition unit to acquire an image of the target human body to obtain an auxiliary acquisition image, collecting the main acquisition images to obtain a main acquisition image set, and collecting the auxiliary acquisition images to obtain an auxiliary acquisition image set, wherein the main acquisition images in the main acquisition image set correspond to the auxiliary acquisition images in the auxiliary acquisition image set in a one-to-one manner; Acquire a main detection image from the main acquisition image set, and construct a pixel intensity histogram based on the acquired main detection image; The total number of pixels and the number of pixel points are obtained according to the pixel intensity histogram, and the average brightness of the pixels is calculated using the total number of pixels and the number of pixel points. The calculation formula of the average brightness of the pixels is as follows: in, Refers to the average brightness of pixels, Refers to the preset grayscale parameters, The grayscale parameter is The number of pixels when Refers to the total number of pixels; The posture video data is obtained according to the average brightness of the pixels, the main detection image and the auxiliary acquisition image set.

[0027] To be clear, the acquisition frequency refers to the manually set frequency used to acquire images of the target human body; the acquisition time refers to the manually set time used to acquire images of the target human body; the primary acquired image refers to the image obtained after the primary acquisition unit acquires images of the target human body; the primary acquisition signal refers to the signal generated by the primary acquisition unit after acquiring images of the target human body; the auxiliary acquired image refers to the image obtained after the auxiliary acquisition unit acquires images of the target human body after receiving the primary acquisition signal; the primary acquired image set refers to the set of all primary acquired images within the acquisition time; the auxiliary acquired image set refers to the set of all auxiliary acquired images within the acquisition time. The primary detection image refers to the first image in the primary acquisition image sequence; the primary acquisition image sequence refers to the sequence obtained by sorting the primary detection images in the primary detection image set from the front to the back according to acquisition time; the pixel intensity histogram refers to a histogram with the grayscale parameter on the horizontal axis and the number of pixels on the vertical axis. The grayscale parameter refers to the parameter used to represent the grayscale value. For example, a grayscale parameter of 1 indicates that the grayscale value of the pixel is 1, and a grayscale parameter of 90 indicates that the grayscale value of the pixel is 90. The total number of pixels refers to the total number of pixels in the main detection image. The number of pixels refers to the number of pixels corresponding to the grayscale parameter. For example, if the grayscale parameter is 2, the number of pixels corresponding to the grayscale parameter 2 is 20, which means that the number of pixels with a grayscale value of 2 is 20. The average pixel brightness is calculated using the total number of pixels and the number of pixels, and is used to reflect the brightness of the main detection image.

[0028] In detail, the step of obtaining the posture video data based on the average pixel brightness, the primary detection image, and the auxiliary acquisition image set includes: Compare the average pixel brightness with a preset brightness threshold; If it is determined that the average pixel brightness is less than the brightness threshold, a corresponding auxiliary adjustment image is obtained from the auxiliary acquisition image set, and the corresponding auxiliary adjustment image is pseudo-color mapped to obtain a mapping acquisition image, wherein the corresponding auxiliary adjustment image is an auxiliary acquisition image corresponding to the main detection image; Converting the mapping acquisition image to obtain a brightness channel image, and calculating the mapping brightness value according to the brightness channel image; The original pixel value is obtained according to the main detection image, and the weighted adjustment brightness value is calculated using the original pixel value and the mapped brightness value, wherein the calculation formula of the weighted adjustment brightness value is as follows: in, Refers to the image horizontal axis parameter , the image vertical axis parameters are The weighted adjustment brightness value is Refers to the preset weighting parameters, Refers to the preset image horizontal axis parameters, Refers to the preset image vertical axis parameters, Refers to the image horizontal axis parameter , the image vertical axis parameters are The mapped brightness value when Refers to the image horizontal axis parameter , the image vertical axis parameters are The original pixel value at time ; Performing brightness replacement on the main detection image using the weighted adjusted brightness value to obtain a target main image; The posture video data is acquired according to the main acquisition image set, the auxiliary acquisition image set and the target main image.

[0029] It can be explained that the brightness threshold refers to a manually set threshold used to determine the average brightness of pixels. When the average brightness of pixels is less than the brightness threshold, it indicates that the average brightness of pixels is too low and needs to be adjusted. The corresponding auxiliary adjustment image refers to the auxiliary acquisition image in the auxiliary acquisition image set corresponding to the main detection image. The mapping acquisition image refers to the image obtained by converting the corresponding auxiliary adjustment image into a pseudo-color image. The brightness channel image refers to the image obtained by converting the mapping acquisition image into a grayscale image. Converting the corresponding auxiliary adjustment image into a pseudo-color image is a prior art method and will not be further described here. The mapped brightness value refers to the grayscale value of the pixel in the brightness channel map, the original pixel value refers to the grayscale value of the pixel in the main detection image, the weighted adjustment brightness value refers to the value obtained after adjusting the original pixel value according to the weighting parameter and the mapped brightness value, the weighting parameter refers to a manually set parameter used to adjust the degree of influence of the mapped brightness value on the weighted adjustment brightness value. The larger the weighting parameter, the greater the influence of the mapped brightness value on the weighted adjustment brightness value. The image horizontal axis parameter refers to the parameter used to represent the horizontal axis of the pixel in the main detection image and the brightness channel map, and the image vertical axis parameter refers to the parameter used to represent the vertical axis of the pixel in the main detection image and the brightness channel map. For example, if the image horizontal axis parameter is 1 and the image vertical axis parameter is 2, the coordinates of the pixel corresponding to the image horizontal axis parameter and the image vertical axis parameter are (1,2). The target main image refers to an image obtained by replacing the grayscale values of the pixels in the main detection image using a weighted adjustment brightness value. Acquiring posture video data based on the main acquisition image set, the auxiliary acquisition image set and the target main image refers to acquiring a target main image sequence using the main acquisition image set and the auxiliary acquisition image set according to the method and steps for acquiring the target main image, and confirming the target main image sequence as posture video data. The target main image sequence refers to a sequence composed of the target main images of all the main acquisition images in the main acquisition image set in the order of acquisition.

[0030] S3. Extracting a first frame of posture image from the posture video data, performing detection based on the extracted first frame of posture image to obtain a preliminary detection frame.

[0031] Interpretably, the first-frame pose image refers to the first image in the target main image sequence.

[0032] In detail, the detection is performed based on the extracted first frame posture image to obtain a preliminary detection frame, including: Inputting the first-frame pose image into a pre-built detection model, and using the detection model to detect the input first-frame pose image to obtain a candidate frame set and a candidate score set, wherein the candidate frames in the candidate frame set correspond one-to-one to the candidate scores in the candidate score set; Extracting candidate scores from the candidate score set in sequence, and comparing the candidate scores with a preset score threshold; If the candidate score is less than the score threshold, the extracted candidate score is removed from the candidate score set to obtain an updated score set, the updated score set is used to update the candidate score set, and the updated candidate score set is used to return to the above step of sequentially extracting candidate scores from the candidate score set until the candidate score set is empty; If the candidate score is not less than the score threshold, the extracted candidate score is input into the pre-constructed target score set, and the extracted candidate score is removed from the candidate score set to obtain an updated score set, the candidate score set is updated using the updated score set, and the updated candidate score set is used to return to the above step of sequentially extracting candidate scores from the candidate score set until the candidate score set is an empty set, wherein the target score set is initially an empty set; A target candidate frame set is matched according to the target score set, and a preliminary detection frame is obtained according to the target candidate frame set.

[0033] Interpretably, the detection model refers to a model that receives the first-frame pose image and outputs a set of candidate boxes and a set of candidate scores. Optionally, the detection model is a combination of the lightweight MobileNet network and the Single Shot MultiBoxDetector. The candidate box set refers to the set of candidate boxes output by the detection model after analyzing the first-frame pose image. The candidate box refers to the box output by the detection model that represents the target human body in the first-frame pose image. The candidate score set refers to the set of candidate scores. The candidate score refers to the confidence score of the candidate box output by the detection model. The score threshold refers to the manually set threshold used to determine the candidate score. The updated score set refers to the set of candidate scores remaining in the candidate score set after the extracted candidate score is removed from the candidate score set. The target score set refers to the set of target scores. The target score refers to the candidate score greater than the score threshold. The target candidate box set refers to the set of target candidate boxes. The target candidate box refers to the candidate box in the candidate box set corresponding to the target score.

[0034] In detail, obtaining a preliminary detection frame based on the target candidate frame set includes: Extract target candidate frames from the target candidate frame set in sequence, calculate areas of the extracted target candidate frames to obtain candidate frame areas, and aggregate the candidate frame areas to obtain a candidate frame area set, wherein the areas of the candidate frames in the candidate frame area set correspond one-to-one to the target candidate frames in the target candidate frame set; A maximum candidate frame area is screened from the candidate frame area set, a maximum candidate frame is matched according to the screened maximum candidate frame area, and the maximum candidate frame is confirmed as a preliminary detection frame.

[0035] Explainably, the candidate box area refers to the area of the target candidate box, the candidate box area set is a set composed of the candidate box areas, the maximum candidate box area refers to the largest candidate box area in the candidate box area set, and the maximum candidate box refers to the candidate box matched in the target candidate box set based on the maximum candidate box area.

[0036] S4. Generate a heat map based on the posture video data to obtain a posture video heat map.

[0037] In detail, generating a heat map based on the posture video data to obtain a posture video heat map includes: performing grayscale normalization on the posture video data to obtain a normalized image sequence; Extracting a first image and a second image in sequence from the normalized image sequence, removing the first image from the normalized image sequence to obtain a removed image sequence, performing calculations based on the extracted first image and second image to obtain a pixel-level absolute difference, and obtaining a pixel difference map based on the pixel-level absolute difference, wherein the first image is the first image in the normalized image sequence, and the second image is an image adjacent to the first image in the normalized image sequence; Using the eliminated image sequence to update the normalized image sequence, and returning to the above steps of sequentially extracting the first image and the second image from the normalized image sequence according to the updated normalized image sequence, until the number of images in the normalized image sequence is 1; A posture video heat map is obtained according to the pixel difference map.

[0038] It can be explained that the normalized image sequence refers to a sequence composed of normalized images, the normalized image refers to the image obtained after normalizing the grayscale value of the target main image in the target main image sequence, the grayscale value normalization refers to obtaining the maximum grayscale value and the minimum grayscale value of each target main image in the target main image sequence, and using the maximum grayscale value and the minimum grayscale value for normalization, the maximum grayscale value refers to the maximum grayscale value in the target main image, and the minimum grayscale value refers to the minimum grayscale value in the target main image. The culled image sequence refers to a sequence composed of the remaining culled images except the first image in the normalized image sequence. The pixel difference map refers to an image composed of the pixel-level absolute differences of two adjacent normalized images in the normalized image sequence. The pixel difference map is used to reflect the changes in the grayscale values of two adjacent normalized images in the normalized image sequence. The grayscale value changes are reflected by the pixel-level absolute differences. The larger the pixel-level absolute difference, the greater the grayscale value changes of the two adjacent normalized images. The pixel-level absolute difference refers to the absolute difference in the grayscale values at the same position of the two adjacent normalized images. For example, the grayscale value of the upper left vertex of the first image is 20, and the grayscale value of the upper left vertex of the second image is 30, then the pixel-level absolute difference is 10. Obtaining a posture video heat map according to the pixel difference map refers to accumulating the pixel-level absolute differences of the corresponding positions of all pixel difference maps in the normalized image sequence to obtain a posture video heat map. For example, assuming that there are 20 normalized images in the normalized image sequence, there are a total of 19 pixel difference maps. The pixel-level absolute differences at all the same positions in the 19 pixel difference maps are accumulated to obtain a posture video heat map.

[0039] In detail, the calculation based on the extracted first image and second image to obtain the pixel-level absolute difference includes: Obtain a normalized first grayscale value of the first image, obtain a normalized second grayscale value of the second image, and calculate a pixel-level absolute difference based on the normalized first grayscale value and the normalized second grayscale value, wherein the calculation formula of the pixel-level absolute difference is as follows: in, Refers to the image horizontal axis parameter , the image vertical axis parameters are The absolute difference at the pixel level is The horizontal axis parameter of the first image is , the image vertical axis parameters are The normalized first grayscale value at The horizontal axis parameter of the second image is , the image vertical axis parameters are The normalized second grayscale value at .

[0040] It can be explained that the normalized first grayscale value refers to the grayscale value of the pixel in the first image, and the normalized second grayscale value refers to the grayscale value of the pixel in the second image.

[0041] S5. Adaptively adjust the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame.

[0042] In detail, the adaptively adjusting the preliminary detection frame using the posture video heat map to obtain the adjusted detection frame includes: Obtaining a preliminary horizontal coordinate, a preliminary vertical coordinate, a preliminary frame width, and a preliminary frame height according to the preliminary detection frame; Constructing a posture motion binary map according to the posture video heat map, and obtaining a motion pixel point set according to the preliminary horizontal coordinate, preliminary vertical coordinate, preliminary frame width, preliminary frame height and the posture motion binary map; Obtaining the upper left corner point and the lower right corner point from the moving pixel point set; The preliminary detection frame is adaptively adjusted according to the upper left corner point and the lower right corner point, the preliminary horizontal coordinate, the preliminary vertical coordinate, the preliminary frame width, and the preliminary frame height to obtain an adjusted detection frame.

[0043] It can be explained that the preliminary horizontal coordinate refers to the horizontal coordinate of the upper left corner vertex of the preliminary detection frame, the preliminary vertical coordinate refers to the vertical coordinate of the upper left corner vertex of the preliminary detection frame, the preliminary frame width refers to the width of the preliminary detection frame, and the preliminary frame height refers to the height of the preliminary detection frame. The posture motion binary map is constructed according to the posture video heat map as follows: a motion threshold is set, and the motion threshold is used to determine the grayscale value of the pixel points in the posture video heat map. Pixel points with grayscale values greater than or equal to the motion threshold are identified as moving pixels, and pixels with grayscale values less than the motion threshold are identified as stationary pixels, completing the construction of the posture motion binary map. The motion threshold refers to an artificially set threshold for determining the grayscale value of the pixel points in the posture video heat map. Obtaining the motion pixel point set according to the preliminary horizontal coordinate, preliminary vertical coordinate, preliminary frame width, preliminary frame height and posture motion binary map refers to obtaining a set of motion pixel points contained in the posture motion binary map by the preliminary detection frame. The upper left corner point refers to the pixel point closest to the upper left corner vertex of the preliminary detection frame in the moving pixel point set, and the lower right corner point refers to the pixel point closest to the lower right corner vertex of the preliminary detection frame in the moving pixel point set.

[0044] Specifically, the adaptively adjusting the preliminary detection frame according to the upper left corner point and the lower right corner point, the preliminary horizontal coordinate, the preliminary vertical coordinate, the preliminary frame width, and the preliminary frame height to obtain the adjusted detection frame includes: The upper left corner point, the lower right corner point, the preliminary horizontal coordinate and the preliminary vertical coordinate are globally mapped to obtain a global upper left corner point and a global lower right corner point, wherein the global upper left corner point is as follows: in, Refers to the global upper left corner, Refers to the initial horizontal axis, Refers to the horizontal coordinate of the upper left corner point, Refers to the initial vertical coordinate, Refers to the vertical coordinate of the upper left corner; The global lower right corner point is as follows: in, Refers to the global lower right corner, Refers to the horizontal coordinate of the lower right corner point, Refers to the vertical coordinate of the lower right corner; The adjustment frame width and the adjustment frame height are obtained according to the global upper left corner point and the global lower right corner point, and the adjustment frame width and the adjustment frame height are used to replace the preliminary frame width and the preliminary frame height of the preliminary detection frame to obtain the adjusted detection frame.

[0045] To be clear, the global upper left corner refers to the coordinate point obtained by mapping the upper left corner point to the coordinate system of the first frame pose image. The global lower right corner refers to the coordinate point obtained by mapping the lower right corner point to the coordinate system of the first frame pose image. The coordinate system of the first frame pose image refers to the coordinate system with the upper left corner vertex of the first frame pose image as the origin, the positive horizontal axis is directly to the right of the upper left corner vertex, and the positive vertical axis is directly below the upper left corner vertex. The adjustment box width refers to the width of the adjustment detection box calculated based on the global upper left corner point and the global lower right corner point. The adjustment box height refers to the height of the adjustment detection box calculated based on the global upper left corner point and the global lower right corner point.

[0046] S6. Obtain a detection main image and a detection auxiliary image from the posture video data according to the adjusted detection frame, and input the obtained detection main image and detection auxiliary image into a pre-built key point network to obtain a main prediction key point set and an auxiliary prediction key point set.

[0047] Explainably, detecting the primary image refers to adjusting the image contained in the detection box within the target primary image in the target primary image sequence, and detecting the auxiliary image refers to adjusting the image contained in the detection box within the auxiliary acquired images in the auxiliary acquired image set. Optionally, the keypoint network is Lite-HRNet-18, a lightweight high-resolution pose network. The primary predicted keypoint set refers to the set of primary predicted keypoints, which are the keypoints output by the keypoint network based on detecting the primary image. The auxiliary predicted keypoint set refers to the set of auxiliary predicted keypoints, which are the keypoints output by the keypoint network based on detecting the auxiliary image. Each primary predicted keypoint and each auxiliary predicted keypoint in the primary and auxiliary predicted keypoint sets have corresponding keypoint confidences. The keypoint confidences refer to the confidences of the primary and auxiliary predicted keypoints, and the primary and auxiliary predicted keypoints have a one-to-one correspondence.

[0048] S7. Perform recognition based on the primary prediction key point set and the auxiliary prediction key point set to obtain a recognition key point set, and complete human body key point recognition based on the recognition key point set.

[0049] Explainably, identification based on the main prediction key point set and the auxiliary prediction key point set refers to comparing the key point confidences of the main prediction key point set and the auxiliary prediction key point set that correspond to each other, and retaining the key point with the larger key point confidence between the two, until all the key points in the main prediction key point set and the auxiliary prediction key point set are compared to obtain the identified key point set.

[0050] The present invention solves the problems described in the background technology. First, a key point recognition instruction is received, and a data acquisition unit is started according to the received key point recognition instruction. The data acquisition unit integrates a main acquisition unit and an auxiliary acquisition unit. The main acquisition unit and the auxiliary acquisition unit are connected by a hardware trigger line to ensure that the main acquisition unit and the auxiliary acquisition unit are almost simultaneously acquiring data, providing image data at the same time for subsequent operations. The overlapping area between the main acquisition unit and the auxiliary acquisition unit is 60°. The 60° overlapping area ensures that the target human body is always in the common visible range of the main acquisition unit and the auxiliary acquisition unit, reducing the omission of key points caused by the blind spot of a single perspective, and providing a reliable data basis for the subsequent acquisition of posture video data; secondly, the main acquisition unit and the auxiliary acquisition unit are used to acquire data on the target human body to obtain posture video data, and the main acquisition unit is acquired according to the image acquired by the auxiliary acquisition unit. The image of the set is adjusted to obtain posture video data, which improves the integrity and clarity of the target human body in the posture video data; then, the first frame posture image is extracted from the posture video data, and a preliminary detection frame is generated based on the first frame posture image. The preliminary detection frame is used to confirm the approximate position of the target human body, and the foundation is laid for the subsequent adjustment of the preliminary detection frame; further, the preliminary detection frame is adaptively adjusted to obtain an adjusted detection frame, and the preliminary detection frame is adjusted according to the posture video heat map to obtain the optimal detection frame representing the target human body, thereby improving the efficiency and accuracy of key point detection; finally, the detection main image and the detection auxiliary image are obtained according to the adjusted detection frame, and the obtained detection main image and detection auxiliary image are input into the key point network to obtain the main prediction key point set and the auxiliary prediction key point set, and the main prediction key point set and the auxiliary prediction key point set are compared to finally obtain the recognition key point set. Therefore, the present invention can improve the accuracy of human key point recognition.

[0051] like Figure 2 , which is a functional module diagram of a human key point recognition system based on posture video data fusion provided by an embodiment of the present invention.

[0052] The human body key point recognition system 100 based on posture video data fusion described in the present invention can be installed in an electronic device. Depending on the functionality implemented, the human body key point recognition system 100 based on posture video data fusion can include a unit startup module 101, a preliminary detection module 102, a frame adjustment module 103, and a key recognition module 104. The modules described in the present invention, also referred to as units, refer to a series of computer program segments that can be executed by an electronic device processor and can perform a fixed function, and are stored in the electronic device's memory.

[0053] The unit startup module 101 is configured to receive a key point recognition instruction and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; The preliminary detection module 102 is used to collect data of a preset target human body using the main collection unit and the auxiliary collection unit to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; The frame adjustment module 103 is used to generate a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; The key recognition module 104 is configured to obtain a detection primary image and a detection secondary image from the posture video data according to the adjusted detection frame, and input the obtained detection primary image and detection secondary image into a pre-built key point network to obtain a primary prediction key point set and a secondary prediction key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

[0054] In detail, each module in the human key point recognition system 100 based on posture video data fusion in the embodiment of the present invention adopts the same method as above when in use. Figure 1 The human body key point recognition method based on posture video data fusion described in the present invention has the same technical means and can produce the same technical effects, so it will not be repeated here.

[0055] like Figure 3 , which is a structural diagram of an electronic device for implementing a method for identifying key points of a human body based on posture video data fusion provided by an embodiment of the present invention.

[0056] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a method program for identifying key points of a human body based on posture video data fusion.

[0057] The memory 11 includes at least one type of readable storage medium, including flash memory, a removable hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 11 includes both the internal storage unit of the electronic device 1 and external storage devices. The memory 11 can be used not only to store application software installed in the electronic device 1 and various data, such as the code of a human key point recognition method based on posture video data fusion, but also to temporarily store data that has been output or is about to be output.

[0058] In some embodiments, the processor 10 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a combination of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core (control unit) of the electronic device, connecting the various components of the electronic device using various interfaces and circuits. It executes programs or modules stored in the memory 11 (e.g., a human body key point recognition method program based on posture video data fusion) and accesses data stored in the memory 11 to perform various functions and process data.

[0059] The bus 12 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus 12 may be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0060] Figure 3 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0061] For example, although not shown, the electronic device 1 may further include a power supply (e.g., a battery) to power various components. Preferably, the power supply may be logically connected to the at least one processor 10 via a power management device, thereby enabling functions such as charge management, discharge management, and power consumption management via the power management device. The power supply may further include any components such as one or more DC or AC power supplies, a recharging device, a power failure detection circuit, a power converter or inverter, and a power status indicator. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not further described here.

[0062] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0063] Optionally, the electronic device 1 may further include a user interface, which may be a display or an input unit (such as a keyboard). Optionally, the user interface may also be a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a display screen or a display unit, and is used to display information processed by the electronic device 1 and to display a visual user interface.

[0064] The human body key point recognition method program based on posture video data fusion stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve the following: Receive a key point recognition instruction, and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; Utilizing the main acquisition unit and the auxiliary acquisition unit to collect data of a preset target human body to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; Generating a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; Obtaining a detection main image and a detection auxiliary image from the posture video data according to the adjusted detection frame, and inputting the obtained detection main image and detection auxiliary image into a pre-built key point network to obtain a main prediction key point set and an auxiliary prediction key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

[0065] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figures 1 to 3 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0066] Furthermore, if the modules / units integrated into the electronic device 1 are implemented as software functional units and sold or used as independent products, they may be stored in a computer-readable storage medium. The computer-readable storage medium may be volatile or non-volatile. For example, the computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0067] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor of an electronic device, the computer program can implement: Receive a key point recognition instruction, and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; Utilizing the main acquisition unit and the auxiliary acquisition unit to collect data of a preset target human body to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; Generating a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; Obtaining a detection main image and a detection auxiliary image from the posture video data according to the adjusted detection frame, and inputting the obtained detection main image and detection auxiliary image into a pre-built key point network to obtain a main prediction key point set and an auxiliary prediction key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

[0068] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, systems and methods can be implemented in other ways. For example, the system embodiments described above are only exemplary, and actual implementations may have other division methods.

[0069] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0070] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0071] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A human body key point recognition method based on posture video data fusion, characterized in that: The method comprises: Receive a key point recognition instruction, and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; Utilizing the main acquisition unit and the auxiliary acquisition unit to collect data of a preset target human body to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; Generating a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; Obtaining a detection main image and a detection auxiliary image from the posture video data according to the adjusted detection frame, and inputting the obtained detection main image and detection auxiliary image into a pre-built key point network to obtain a main prediction key point set and an auxiliary prediction key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

2. The human body key point recognition method based on posture video data fusion as claimed in claim 1, characterized in that: The method of collecting data of a preset target human body by using the main collection unit and the auxiliary collection unit to obtain posture video data includes: Based on a preset acquisition frequency, a preset acquisition time and the main acquisition unit, an image of the target human body is acquired to obtain a main acquisition image and generate a main acquisition signal, wherein a main acquisition signal is generated each time an image is acquired; Using the hardware trigger line to send the main acquisition signal to the auxiliary acquisition unit, and using the auxiliary acquisition unit to acquire an image of the target human body to obtain an auxiliary acquisition image, collecting the main acquisition images to obtain a main acquisition image set, and collecting the auxiliary acquisition images to obtain an auxiliary acquisition image set, wherein the main acquisition images in the main acquisition image set correspond to the auxiliary acquisition images in the auxiliary acquisition image set in a one-to-one manner; Acquire a main detection image from the main acquisition image set, and construct a pixel intensity histogram based on the acquired main detection image; The total number of pixels and the number of pixel points are obtained according to the pixel intensity histogram, and the average brightness of the pixels is calculated using the total number of pixels and the number of pixel points. The calculation formula of the average brightness of the pixels is as follows: in, Refers to the average brightness of pixels, Refers to the preset grayscale parameters, The grayscale parameter is The number of pixels when Refers to the total number of pixels; The posture video data is obtained according to the average brightness of the pixels, the main detection image and the auxiliary acquisition image set.

3. The human body key point recognition method based on posture video data fusion as claimed in claim 2, characterized in that: The acquiring of posture video data according to the average brightness of the pixels, the main detection image, and the auxiliary acquisition image set includes: Compare the average pixel brightness with a preset brightness threshold; If it is determined that the average pixel brightness is less than the brightness threshold, a corresponding auxiliary adjustment image is obtained from the auxiliary acquisition image set, and the corresponding auxiliary adjustment image is pseudo-color mapped to obtain a mapping acquisition image, wherein the corresponding auxiliary adjustment image is an auxiliary acquisition image corresponding to the main detection image; Converting the mapping acquisition image to obtain a brightness channel image, and calculating the mapping brightness value according to the brightness channel image; The original pixel value is obtained according to the main detection image, and the weighted adjustment brightness value is calculated using the original pixel value and the mapped brightness value, wherein the calculation formula of the weighted adjustment brightness value is as follows: in, Refers to the image horizontal axis parameter , the image vertical axis parameters are The weighted adjustment brightness value is Refers to the preset weighting parameters, Refers to the preset image horizontal axis parameters, Refers to the preset image vertical axis parameters, Refers to the image horizontal axis parameter , the image vertical axis parameters are The mapped brightness value when Refers to the image horizontal axis parameter , the image vertical axis parameters are The original pixel value at time ; Performing brightness replacement on the main detection image using the weighted adjusted brightness value to obtain a target main image; The posture video data is acquired according to the main acquisition image set, the auxiliary acquisition image set and the target main image.

4. The human body key point recognition method based on posture video data fusion as claimed in claim 3, characterized in that: The detection is performed based on the extracted first frame posture image to obtain a preliminary detection frame, including: Inputting the first-frame pose image into a pre-built detection model, and using the detection model to detect the input first-frame pose image to obtain a candidate frame set and a candidate score set, wherein the candidate frames in the candidate frame set correspond one-to-one to the candidate scores in the candidate score set; Extracting candidate scores from the candidate score set in sequence, and comparing the candidate scores with a preset score threshold; If the candidate score is less than the score threshold, the extracted candidate score is removed from the candidate score set to obtain an updated score set, the updated score set is used to update the candidate score set, and the updated candidate score set is used to return to the above step of sequentially extracting candidate scores from the candidate score set until the candidate score set is empty; If the candidate score is not less than the score threshold, the extracted candidate score is input into the pre-constructed target score set, and the extracted candidate score is removed from the candidate score set to obtain an updated score set, the candidate score set is updated using the updated score set, and the updated candidate score set is used to return to the above step of sequentially extracting candidate scores from the candidate score set until the candidate score set is an empty set, wherein the target score set is initially an empty set; A target candidate frame set is matched according to the target score set, and a preliminary detection frame is obtained according to the target candidate frame set.

5. The human body key point recognition method based on posture video data fusion as claimed in claim 4, characterized in that: The obtaining of a preliminary detection frame according to the target candidate frame set includes: Extract target candidate frames from the target candidate frame set in sequence, calculate areas of the extracted target candidate frames to obtain candidate frame areas, and aggregate the candidate frame areas to obtain a candidate frame area set, wherein the areas of the candidate frames in the candidate frame area set correspond one-to-one to the target candidate frames in the target candidate frame set; A maximum candidate frame area is screened from the candidate frame area set, a maximum candidate frame is matched according to the screened maximum candidate frame area, and the maximum candidate frame is confirmed as a preliminary detection frame.

6. The human body key point recognition method based on posture video data fusion as claimed in claim 5, characterized in that: Generating a heat map based on the posture video data to obtain a posture video heat map includes: performing grayscale normalization on the posture video data to obtain a normalized image sequence; Extracting a first image and a second image in sequence from the normalized image sequence, removing the first image from the normalized image sequence to obtain a removed image sequence, performing calculations based on the extracted first image and second image to obtain a pixel-level absolute difference, and obtaining a pixel difference map based on the pixel-level absolute difference, wherein the first image is the first image in the normalized image sequence, and the second image is an image adjacent to the first image in the normalized image sequence; Using the eliminated image sequence to update the normalized image sequence, and returning to the above steps of sequentially extracting the first image and the second image from the normalized image sequence according to the updated normalized image sequence, until the number of images in the normalized image sequence is 1; A posture video heat map is obtained according to the pixel difference map.

7. The human body key point recognition method based on posture video data fusion as claimed in claim 6, characterized in that: The calculating based on the extracted first image and second image to obtain the pixel-level absolute difference includes: Obtain a normalized first grayscale value of the first image, obtain a normalized second grayscale value of the second image, and calculate a pixel-level absolute difference based on the normalized first grayscale value and the normalized second grayscale value, wherein the calculation formula of the pixel-level absolute difference is as follows: in, Refers to the image horizontal axis parameter , the image vertical axis parameters are The absolute difference at the pixel level is The horizontal axis parameter of the first image is , the image vertical axis parameters are The normalized first grayscale value at time , The horizontal axis parameter of the second image is , the image vertical axis parameters are The normalized second grayscale value at .

8. The human body key point recognition method based on posture video data fusion as claimed in claim 7, characterized in that: Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame includes: Obtaining a preliminary horizontal coordinate, a preliminary vertical coordinate, a preliminary frame width, and a preliminary frame height according to the preliminary detection frame; Constructing a posture motion binary map according to the posture video heat map, and obtaining a motion pixel point set according to the preliminary horizontal coordinate, preliminary vertical coordinate, preliminary frame width, preliminary frame height and the posture motion binary map; Obtaining the upper left corner point and the lower right corner point from the moving pixel point set; The preliminary detection frame is adaptively adjusted according to the upper left corner point and the lower right corner point, the preliminary horizontal coordinate, the preliminary vertical coordinate, the preliminary frame width, and the preliminary frame height to obtain an adjusted detection frame.

9. The human body key point recognition method based on posture video data fusion as claimed in claim 8, characterized in that: Adaptively adjusting the preliminary detection frame according to the upper left corner point and the lower right corner point, the preliminary horizontal coordinate, the preliminary vertical coordinate, the preliminary frame width, and the preliminary frame height to obtain the adjusted detection frame includes: The upper left corner point, the lower right corner point, the preliminary horizontal coordinate and the preliminary vertical coordinate are globally mapped to obtain a global upper left corner point and a global lower right corner point, wherein the global upper left corner point is as follows: in, Refers to the global upper left corner, Refers to the initial horizontal axis, Refers to the horizontal coordinate of the upper left corner point, Refers to the initial vertical coordinate, Refers to the vertical coordinate of the upper left corner; The global lower right corner point is as follows: in, Refers to the global lower right corner, Refers to the horizontal coordinate of the lower right corner point, Refers to the vertical coordinate of the lower right corner; The adjustment frame width and the adjustment frame height are obtained according to the global upper left corner point and the global lower right corner point, and the adjustment frame width and the adjustment frame height are used to replace the preliminary frame width and the preliminary frame height of the preliminary detection frame to obtain the adjusted detection frame.

10. A human key point recognition system based on posture video data fusion, characterized in that: The system comprises: a unit startup module, configured to receive a key point recognition instruction and start a pre-built data acquisition unit according to the key point recognition instruction, wherein the data acquisition unit includes: a main acquisition unit and an auxiliary acquisition unit, and the main acquisition unit and the auxiliary acquisition unit are connected via a hardware trigger line, and the overlap area between the main acquisition unit and the auxiliary acquisition unit is 60 degrees; A preliminary detection module is used to collect data of a preset target human body using the main acquisition unit and the auxiliary acquisition unit to obtain posture video data; Extracting a first-frame posture image from the posture video data, and performing detection based on the extracted first-frame posture image to obtain a preliminary detection frame; A frame adjustment module is used to generate a heat map based on the posture video data to obtain a posture video heat map; Adaptively adjusting the preliminary detection frame using the posture video heat map to obtain an adjusted detection frame; a key recognition module, configured to obtain a detection primary image and a detection secondary image from the posture video data according to the adjusted detection frame, and input the obtained detection primary image and detection secondary image into a pre-built key point network to obtain a primary predicted key point set and a secondary predicted key point set; Identification is performed based on the main prediction key point set and the auxiliary prediction key point set to obtain an identification key point set, and human body key point recognition is completed according to the identification key point set.

Images