Multi-modal information fusion body test evaluation method, electronic equipment and storage medium

Abstract

The invention discloses a multi-modal information fusion body test evaluation method, electronic equipment and a storage medium, and relates to the technical field of image processing, the multi-modal information fusion body test evaluation method specifically comprises the following steps: detecting a body test target to obtain a plurality of initial image data, and obtaining initial text data corresponding to the body test target; performing fusion processing on the plurality of initial image data to obtain target image data, and determining a body measurement posture score corresponding to the body measurement target according to the target image data; generating fusion feature data based on the target image data, the body measurement posture score and the initial text data; and determining a target body measurement evaluation result according to the fused feature data. By adopting the method and the device, the technical effect of enabling the electronic equipment to generate a body test evaluation result with higher accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-modal information fusion body measurement and evaluation method, electronic device and storage medium. Background Art

[0002] With the continuous development of image information processing technology, image processing functions play an increasingly important role in the evaluation of physical examination items.

[0003] In related technologies, in order to more accurately detect the athletic performance of the physical measurement target, technicians usually set up a camera device to track and shoot the physical measurement target, thereby identifying the physical measurement target's movement posture in the physical measurement item through the captured image data, and then obtaining the physical measurement evaluation result based on the identified movement posture.

[0004] However, since the camera device is easily affected by ambient light when shooting the body measurement target, it is easy for the image data obtained by the camera device to have low quality, which makes it difficult for the electronic device to accurately identify the movement posture adopted by the body measurement target, and thus makes the final generated body measurement evaluation result less accurate. Summary of the invention

[0005] The main purpose of the present application is to provide a physical measurement and assessment method, electronic device and storage medium with multimodal information fusion, aiming to solve the technical problem of low accuracy of physical measurement and assessment results generated by electronic devices in related technologies.

[0006] To achieve the above objectives, the present application proposes a multi-modal information fusion physical measurement evaluation method, comprising:

[0007] Detecting a body measurement target to obtain a plurality of initial image data, and obtaining initial text data corresponding to the body measurement target;

[0008] Performing fusion processing on the plurality of the initial image data to obtain target image data, and determining a physical measurement posture score corresponding to the physical measurement target according to the target image data;

[0009] generating fused feature data based on the target image data, the body measurement posture score and the initial text data;

[0010] The target body measurement evaluation result is determined according to the fused feature data.

[0011] In one embodiment, the step of fusing the plurality of initial image data to obtain target image data comprises:

[0012] Extracting image features corresponding to each of the plurality of initial image data, and generating a plurality of heat maps corresponding to each of the initial image data according to each of the image features;

[0013] The first key points contained in each of the heat maps are determined, and the heat maps are fused based on the first key points to obtain target image data.

[0014] In one embodiment, the step of fusing the heat maps based on the first key points to obtain target image data includes:

[0015] Determining a confidence level of each of the first key points for matching, and determining a fusion weight of each of the first key points for matching according to each of the confidence levels;

[0016] The heat maps are fused based on the fusion weights to obtain target image data.

[0017] In one embodiment, the step of determining the physical measurement posture score corresponding to the physical measurement target according to the target image data includes:

[0018] Identify the actual body measurement posture corresponding to the body measurement target according to the target image data, and determine a standard body measurement posture matching the actual body measurement posture;

[0019] The second key points included in the standard body measurement posture are obtained, and a body measurement posture score corresponding to the body measurement target is determined according to the first key points and the second key points.

[0020] In one embodiment, the step of determining a physical test posture score corresponding to the physical test target according to each of the first key points and each of the second key points includes:

[0021] Determine the actual Euclidean distance between each of the first key points and the matched second key point;

[0022] Determine a target preset posture score that matches each of the actual Euclidean distances, and determine a preset score weight that corresponds to each of the first key points;

[0023] The preset posture scores are processed based on the preset scoring weights to determine the physical measurement posture score corresponding to the physical measurement target.

[0024] In one embodiment, the step of determining the target preset posture score that matches each of the actual Euclidean distances comprises:

[0025] Acquire a plurality of preset Euclidean distances and preset posture scores corresponding to each of the plurality of preset Euclidean distances;

[0026] Screening the plurality of preset Euclidean distances based on the actual Euclidean distances to determine a target preset Euclidean distance that matches each of the actual Euclidean distances;

[0027] According to the preset posture scores corresponding to the target preset Euclidean distances, the target preset posture scores that match the actual Euclidean distances are determined.

[0028] In one embodiment, the step of generating fused feature data based on the target image data, the body measurement posture score and the initial text data comprises:

[0029] Performing conversion processing based on the target image data to obtain a target image vector;

[0030] Performing conversion processing based on the body measurement posture score and the initial text data to obtain a target text vector;

[0031] The target image vector and the target text vector are mapped to a target feature space to fuse and generate fused feature data.

[0032] In one embodiment, before the step of detecting the body measurement target to obtain a plurality of initial image data, the method further includes:

[0033] Acquire a plurality of preset positive sample pairs and negative sample pairs, wherein the positive sample pairs are sample pairs consisting of preset image data and matching preset text data, and the negative sample pairs are sample pairs consisting of preset image data and random preset text data;

[0034] Determine each distance difference generated between each of the positive sample pairs and each of the negative sample pairs, and determine a contrast loss function according to each of the distance differences;

[0035] Based on the contrast loss function, the preset initial body measurement prediction model is adjusted to generate a target body measurement prediction model, so that the target body measurement prediction model processes the multiple initial image data and the initial text data to obtain a target body measurement evaluation result.

[0036] In addition, to achieve the above-mentioned purpose, the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the physical measurement and assessment method of multimodal information fusion as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the physical measurement and evaluation method of multimodal information fusion as described above are implemented.

[0038] The physical measurement and assessment method of multimodal information fusion provided in the embodiment of the present application detects a physical measurement target to obtain multiple initial image data, and obtains initial text data corresponding to the physical measurement target; fuses the multiple initial image data to obtain target image data, and determines the physical measurement posture score corresponding to the physical measurement target according to the target image data; generates fused feature data based on the target image data, the physical measurement posture score and the initial text data; and determines the target physical measurement assessment result according to the fused feature data.

[0039] In this embodiment, when the electronic device is running, it first calls multiple camera devices with different shooting angles to shoot the physical measurement target to obtain multiple initial image data containing the physical measurement target. At the same time, the electronic device reads the storage module configured by itself to obtain the initial text data pre-input by the physical measurement target. After that, the electronic device fuses the multiple initial image data with different perspectives to obtain target image data containing semantic information of different initial image data, and determines the physical measurement posture score of the physical measurement target during the physical measurement process according to the target image data. After that, the electronic device processes the target image data, the physical measurement posture score and the initial text data to obtain fused feature data containing multiple semantic information at the same time. Finally, the electronic device generates a target physical measurement evaluation result that matches the physical measurement target according to the fused feature data.

[0040] In this way, the present application solves the technical problem of low accuracy of physical fitness assessment results generated by electronic devices in the related art. That is, the present application obtains initial image data from multiple angles by shooting, thereby avoiding the situation where a single camera device is easily affected by ambient light during the shooting process. At the same time, the present application obtains target image data by fusing multiple initial image data, and obtains fused feature data by fusing the target image data with the posture score and text data, so as to generate physical fitness assessment results according to the fused feature data. In this way, the electronic device considers factors such as the dynamic motion performance and static body performance of the physical fitness target while generating the physical fitness assessment results, thereby achieving the technical effect of enabling the electronic device to generate more accurate physical fitness assessment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart diagram of the first embodiment of the physical measurement and evaluation method for multimodal information fusion of the present application is provided;

[0044] Figure 2 A schematic diagram of a physical measurement scenario involved in an embodiment of a physical measurement evaluation method for multimodal information fusion of the present application;

[0045] Figure 3 A flowchart diagram of the second embodiment of the physical measurement and evaluation method for multimodal information fusion of the present application is provided;

[0046] Figure 4 This is a schematic diagram of the module structure of a physical measurement and evaluation device for multimodal information fusion according to an embodiment of the present application;

[0047] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the physical measurement and evaluation method for multimodal information fusion in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] In this embodiment, for ease of description, the following description is based on an electronic device that is internally configured with a plurality of cameras with different angles and a model training module, or a mobile terminal, a data storage control terminal, a PC or other terminal connected to an electronic control unit that is matched with the electronic device as the execution entity.

[0052] Based on the above electronic device, the overall concept of the physical measurement and evaluation method of multimodal information fusion of the present application is proposed here.

[0053] With the continuous development of image information processing technology, image processing functions play an increasingly important role in the evaluation of physical test items. In related technologies, in order to more accurately detect the sports performance of the physical test target, technicians usually set up a camera device to track and shoot the physical test target, so as to identify the sports posture of the physical test target in the physical test item through the captured image data, and then obtain the physical test evaluation result based on the identified sports posture. However, since the camera device is easily affected by ambient light during the process of shooting the physical test target, it is easy to have a situation where the image data obtained by the camera device is of low quality, which makes it difficult for the electronic device to accurately identify the sports posture adopted by the physical test target, and thus makes the final generated physical test evaluation result less accurate. In addition, since the camera device can only capture the dynamic motion information of the physical test target, but cannot obtain static body information such as physical fitness, cardiopulmonary function, etc., the physical test evaluation result generated by the above method can only partially evaluate the dynamic physical test results of the physical test target, and cannot comprehensively evaluate the physical test target.

[0054] In response to the above phenomenon, the present application provides a physical measurement and assessment method with multimodal information fusion, which includes: detecting a physical measurement target to obtain multiple initial image data, and obtaining initial text data corresponding to the physical measurement target; fusing the multiple initial image data to obtain target image data, and determining a physical measurement posture score corresponding to the physical measurement target based on the target image data; generating fused feature data based on the target image data, the physical measurement posture score and the initial text data; and determining a target physical measurement assessment result based on the fused feature data.

[0055] In this way, the present application solves the technical problem of low accuracy of physical fitness assessment results generated by electronic devices in the related art. That is, the present application obtains initial image data from multiple angles by shooting, thereby avoiding the situation where a single camera device is easily affected by ambient light during the shooting process. At the same time, the present application obtains target image data by fusing multiple initial image data, and obtains fused feature data by fusing the target image data with the posture score and text data, so as to generate physical fitness assessment results according to the fused feature data. In this way, the electronic device considers factors such as the dynamic motion performance and static body performance of the physical fitness target while generating the physical fitness assessment results, thereby achieving the technical effect of enabling the electronic device to generate more accurate physical fitness assessment results.

[0056] Based on the overall concept of the physical measurement and evaluation method of multimodal information fusion of the present application, the embodiment of the present application provides a physical measurement and evaluation method of multimodal information fusion, referring to Figure 2 , Figure 2This is a flow chart of the first embodiment of the physical test and assessment method for multimodal information fusion of the present application. In this embodiment, the physical test and assessment method for multimodal information fusion includes steps S10 to S40:

[0057] Step S10: Detecting a body measurement target to obtain a plurality of initial image data, and obtaining initial text data corresponding to the body measurement target.

[0058] Step S20: performing fusion processing on the plurality of the initial image data to obtain target image data, and determining the physical measurement posture score corresponding to the physical measurement target according to the target image data.

[0059] It should be noted that the initial image data is a color RGB image including the physical measurement target obtained by the electronic device through a camera device when the physical measurement target is being measured; it is understandable that the initial image data can be other types of images such as infrared images in addition to color RGB images, and the present application does not limit the image type of the initial image data. In addition, the initial text data can specifically be basic physical fitness text data and / or cardiopulmonary function text data input by the physical measurement target. It is understandable that the specific content of the initial text data can be set by the technician according to the needs of the physical measurement, and the present application does not limit this. Similarly, the initial text data is input into the electronic device by the physical measurement target before the physical measurement project is carried out, and is stored in the storage module configured by the electronic device itself. In addition, the target image data is a fused image data containing the semantic information of each of the multiple initial image data, that is, the semantic information of each of the multiple initial image data can be obtained through the target image data. In addition, the physical test posture score is used to reflect the standard degree of the physical test posture adopted by the physical test target during the physical test. For example, when the physical test item is pull-ups, the physical test posture score can be used to determine whether the pull-ups of the physical test target are standard. Similarly, when the physical test item is push-ups, the physical test posture score can be used to determine whether the push-ups of the physical test target are standard. It can be understood that this application does not limit the specific content of the physical test items.

[0060] In this embodiment, when the electronic device is running, it first calls multiple camera devices with different shooting angles configured by itself to shoot the physical measurement target, so as to capture multiple initial image data including the physical measurement target. At the same time, the electronic device reads the storage module configured by itself to obtain the initial text data pre-stored by the physical measurement target before the physical measurement. After that, the electronic device inputs the obtained multiple initial image data and initial text data into the target physical measurement prediction model configured by itself, and the target physical measurement prediction model processes the multiple initial image data so that the semantics contained in the multiple initial image data can affect each other, so as to fuse the target image data. The target physical measurement prediction model then identifies the target image data to determine the actual physical measurement posture adopted by the physical measurement target during the physical measurement process, and determines the physical measurement posture score corresponding to the actual physical measurement posture.

[0061] For example, see Figure 2 , Figure 2 This is a schematic diagram of a physical measurement scenario involved in an embodiment of a physical measurement evaluation method for multimodal information fusion of the present application. When the electronic device is running, it first calls its own configured Figure 2 The multiple camera devices with different viewing angles shown shoot the physical measurement target to capture multiple color RGB images containing the physical measurement target. At the same time, the electronic device reads the storage module configured by itself to obtain the basic physical fitness text data and cardiopulmonary function test text data of the physical measurement target input into the storage module. After that, the electronic device inputs the obtained multiple color RGB images, basic physical fitness text data, and cardiopulmonary function test text data into the target physical measurement prediction model configured by itself. The target physical measurement prediction model processes the multiple color RGB images to fuse the semantic information contained in each of the multiple color RGB images, thereby generating a posture estimation heat map containing the semantic information of each of the multiple RGB images. The target physical measurement prediction model then identifies the posture estimation heat map to determine the actual physical measurement posture adopted by the detection target during the physical measurement process, and determines the physical measurement posture score that matches the actual physical measurement posture.

[0062] In this way, the electronic device can capture multiple image data with different perspectives and containing the physical measurement target through multiple camera devices with different perspectives, and identify the actual physical measurement posture adopted by the physical measurement target in the physical measurement item, thereby determining the physical measurement posture score corresponding to the actual physical measurement posture, and then generate a more comprehensive physical measurement evaluation result based on the physical measurement posture score.

[0063] In a feasible implementation manner, the above step S20 of "merging the plurality of initial image data to obtain target image data" may specifically include steps S210 to S220:

[0064] Step S210: extracting image features corresponding to each of the plurality of initial image data, and generating heat maps corresponding to each of the plurality of initial image data according to each of the image features.

[0065] Step S220: determining the first key points contained in each of the heat maps, and fusing the heat maps based on the first key points to obtain target image data.

[0066] It should be noted that the first key point is the location information of the key parts of the body measurement target (such as head, shoulder, elbow, wrist, hip, knee, ankle) in the heat map. In addition, the heat map is used to indicate the probability of each pixel belonging to a key point, that is, the position of the pixel with the highest probability in the heat map is the key point.

[0067] In this embodiment, after obtaining multiple initial image data, the target body measurement estimation model first extracts the image features of each of the multiple initial image data, and determines the semantic information that matches each of the multiple initial image data based on each image feature. The target liveness prediction model then determines the key point probability that each pixel point matches based on each semantic information, and generates a heat map containing different colors that matches each initial image data based on each key point probability. After that, the target body measurement estimation model identifies each thermal map to determine each first key point contained in each thermal map, and fuses each thermal map based on each first key point, so that each thermal map can obtain the semantic information contained in other thermal maps, and then generate fused image data containing the semantic information corresponding to all thermal maps.

[0068] Exemplarily, for example, after obtaining multiple color RGB images, the target liveness prediction model first processes the multiple color RGB images to extract image features that match each of the multiple color RGB images. The target liveness prediction model then determines the probability of pixels contained in each of the multiple color RGB images being at key points based on each image feature, and generates a heat map containing multiple colors that matches each of the multiple color RGB images based on each probability. Thereafter, the target liveness prediction model identifies the first 2D key points contained in each heat map based on the colors contained in each heat map, and fuses each heat map based on each first 2D key point to generate a posture estimation heat map containing semantic information contained in all color RGB images.

[0069] In addition, in this embodiment and another embodiment, after obtaining multiple color RGB images, if the target live prediction model identifies that part of the body measurement target is occluded by other targets in the image based on the image features, the target live prediction model uses a sub-pixel difference key point extraction algorithm to estimate the occluded part of the body measurement target to determine the first 2D key point corresponding to the occluded part. It can be understood that the specific process of estimating the first 2D key point by using the sub-pixel difference key point extraction algorithm is a prior art, so it will not be repeated here.

[0070] In this way, the electronic device can fuse image data from multiple different perspectives to obtain fused image data containing the semantic information of all the image data, and then identify the actual physical measurement posture adopted by the physical measurement target in the physical measurement item based on the fused image data, so as to determine the physical measurement posture score corresponding to the actual physical measurement posture, and generate a more comprehensive physical measurement evaluation result based on the physical measurement posture score.

[0071] In a feasible implementation manner, the above step S20 may specifically include steps S221 to S222:

[0072] Step S221: determining the confidence level of each of the first key points in the match, and determining the fusion weight of each of the first key points in the match according to each of the confidence levels.

[0073] Step S222: performing fusion processing on each of the thermal maps based on each of the fusion weights to obtain target image data.

[0074] In this embodiment, after extracting the first key points contained in each of the multiple thermal maps, the target body measurement prediction model first identifies each first key point to determine the confidence level of each match of the first key point. The target body measurement prediction model then assigns fusion weights to each first key point according to each confidence level. Thereafter, the target body measurement prediction model fuses the multiple thermal maps according to the preset fusion factor coefficient and each fusion weight, thereby generating fused image data containing the semantic information of each of the multiple thermal maps.

[0075] Exemplarily, for example, after extracting the first 2D key points contained in each of the multiple color RGB images, the target liveness prediction model first determines the confidence corresponding to each of the multiple first 2D keys, and assigns a corresponding fusion weight to each first 2D key according to the confidence. After that, the target body measurement estimation model fuses the multiple heat maps according to each fusion weight, so that the heat map with a higher fusion weight has a greater impact on the heat maps of other surrounding fusion weight angles, thereby obtaining a posture estimation heat map containing the semantic information contained in each color RGB image and accurate first 2D key points.

[0076] It should be noted that, in this embodiment and another embodiment, after determining multiple first 2D key points, the target volume measurement estimation model can determine the confidence levels corresponding to the multiple first 2D key points based on whether the parts where the multiple first 2D key points are located are blocked by other units in the image. Similarly, the target volume measurement estimation model can also determine the confidence levels corresponding to the multiple first 2D key points based on whether the parts where the multiple first 2D key points are located are complete and clear. It can be understood that the present application does not impose any restrictions on the specific values ​​of the confidence levels of the respective first 2D key points.

[0077] In addition, in this embodiment and another embodiment, after determining the confidence of each first 2D key point, the target body measurement estimation model can query a weight mapping table containing multiple preset confidences and preset assigned weights that match each of the multiple preset confidences, thereby determining the fusion weights that match each of the confidences. It can be understood that the present application also does not impose any restrictions on the specific values ​​of the assigned fusion weights.

[0078] In this way, the electronic device can fuse image data from multiple different perspectives to obtain fused image data containing the semantic information of all the image data, and then identify the actual physical measurement posture adopted by the physical measurement target in the physical measurement item based on the fused image data, so as to determine the physical measurement posture score corresponding to the actual physical measurement posture, and generate a more comprehensive physical measurement evaluation result based on the physical measurement posture score.

[0079] In a feasible implementation manner, the above step S20 of "determining the physical measurement posture score corresponding to the physical measurement target according to the target image data" may specifically include steps S230 to S240:

[0080] Step S230: identifying the actual body measurement posture corresponding to the body measurement target according to the target image data, and determining a standard body measurement posture matching the actual body measurement posture.

[0081] Step S240: Acquire each second key point included in the standard body measurement posture, and determine the body measurement posture score corresponding to the body measurement target according to each first key point and each second key point.

[0082] In this embodiment, after fusing the above-mentioned fused image data, the target body measurement prediction model first identifies the fused image data to determine the actual body measurement posture adopted by the body measurement target in the body measurement item based on the fused image data. At the same time, the target body measurement prediction model queries the standard body measurement posture that matches the actual body measurement posture. After that, the target body measurement prediction model determines the second key point contained in the standard body measurement posture, and determines the corresponding body measurement posture score of the body measurement target when adopting the actual body measurement posture based on each first key point and each second key point.

[0083] Exemplarily, for example, after obtaining the posture estimation heat map, the target body measurement prediction model first identifies the posture estimation heat map to determine the actual body measurement posture taken by the body measurement target in the body measurement item through the posture estimation heat map. If the target body measurement prediction model detects that the actual body measurement posture is a push-up posture, the target body measurement prediction model queries a preset standard posture database to determine a matching standard push-up posture. After that, the target body measurement prediction model reads the posture estimation heat map to determine the elbows, waist, and other parts of the body measurement target when the body measurement target adopts a push-up posture. The first 2D key points corresponding to the legs and other positions respectively. At the same time, the target body measurement prediction model analyzes the standard push-up posture to determine the second 2D key points corresponding to the elbows, waist, legs and other parts in the standard push-up posture. The target body measurement prediction model determines the elbows, waist, legs and other positions of the body measurement target based on the first 2D key points and the second 2D key points, and the differences between the elbows, waist, legs and other positions in the standard push-up posture, and then obtains the body measurement posture score corresponding to the push-up posture adopted by the body measurement target based on the differences.

[0084] Similarly, if the target body measurement prediction model detects that the actual body measurement posture is a pull-up posture, the target body measurement prediction model queries the above-mentioned standard posture database to determine the matching standard pull-up posture. After that, the target body measurement prediction model determines the first 2D key points corresponding to the elbows, waist, legs and other positions of the body measurement target when doing pull-ups through the posture estimation heat map. At the same time, the target body measurement prediction model analyzes the standard pull-up posture to determine the second 2D key points corresponding to the elbows, waist, legs and other positions in the standard pull-up posture. Finally, the target body measurement prediction model determines the differences between the elbows, waist, legs and other positions of the body measurement target and the elbows, waist, legs and other positions in the standard pull-up posture based on each first 2D key point and each second 2D key point, and then obtains the body measurement posture score corresponding to the pull-up adopted by the body measurement target based on each difference.

[0085] It should be noted that, in addition to push-ups and pull-ups, the actual physical test posture can also be other postures such as running. It is understandable that the present application does not limit the specific type of the actual physical test posture. In addition, the actual physical test posture can be the posture of the physical test target at any stage of the physical test item when completing a physical test item (for example, the actual physical test posture can be the posture of the physical test target in the push-up force stage, or it can be the posture of the physical test target when the body is at the highest point when doing push-ups, or it can be the posture of the physical test target when the body is in the downward movement stage when doing push-ups). It is understandable that the present application also does not limit the stage of the actual physical test posture in the physical test item.

[0086] In this way, the electronic device can identify the actual physical test posture adopted by the physical test target in the physical test item, thereby determining the physical test posture score corresponding to the actual physical test posture, and then generate a more comprehensive physical test evaluation result based on the physical test posture score.

[0087] In a feasible implementation manner, the step of "determining the physical test posture score corresponding to the physical test target according to each of the first key points and each of the second key points" in the above step S240 may specifically include steps S241 to S243:

[0088] Step S241: determining the actual Euclidean distance between each of the first key points and the matched second key points.

[0089] Step S242: Determine the target preset posture score that matches each of the actual Euclidean distances, and determine the preset score weight corresponding to each of the first key points.

[0090] Step S243: Processing each of the preset posture scores based on each of the preset scoring weights to determine the physical measurement posture score corresponding to the physical measurement target.

[0091] In this embodiment, after determining each first key point and each second key point, the target body measurement prediction model first detects the Euclidean distance between each first key point and the matching second key point. Then, the target body measurement prediction model determines the target preset posture scores that each matches the multiple preset Euclidean distances. At the same time, the target body measurement prediction model determines the preset score weights that each matches the multiple first key points. Finally, the target body measurement prediction model performs calculations based on the preset posture scores of each target and the preset score weights that each matches, so as to obtain the body measurement posture score corresponding to the actual body measurement posture adopted by the body measurement target.

[0092] Exemplarily, for example, after determining each first 2D key point and each second 2D key point, the target body measurement prediction model first extracts the first 2D key points corresponding to the elbows, waist, legs and other parts of the body measurement target, and determines the actual Euclidean distance between each first 2D key point and the matching second 2D key point. After that, the target body measurement prediction model reads the above-mentioned storage module to obtain a preset score mapping table, and then queries the score mapping table based on each actual Euclidean distance to determine the target preset posture score that matches the actual Euclidean distance corresponding to each part of the elbows, waist, legs and other parts. When the target body measurement prediction model determines that the preset score weight of the first 2D key point corresponding to the elbow is 33%, and the preset score weight of the first 2D key point corresponding to the waist is 33%, and the preset score weight of the first 2D key point corresponding to the leg is 34%. The target body measurement prediction model processes the preset posture score of each target based on the preset score weights to obtain the body measurement posture score when the body measurement target adopts actual body measurement postures such as push-ups / pull-ups: 33%*elbow target preset posture score+33%*waist target preset posture score+34%*leg target preset posture score.

[0093] It should be noted that, in this embodiment and another embodiment, after determining each first 2D key point and each second 2D key point, the target body measurement prediction model can also only determine the target first 2D key point corresponding to the elbow / waist / leg of the body measurement target, and determine the target second 2D key point that matches the target first 2D key point under the standard body measurement posture. After that, the target body measurement prediction model determines the actual Euclidean distance between the elbow of the body measurement target and the elbow in the standard push-up posture based on the target first 2D key point and the target second 2D key point. Finally, the target body measurement prediction model queries the above-mentioned score mapping table based on the actual Euclidean distance to determine the target preset posture score that matches the actual Euclidean distance, and directly determines the target preset posture score as the body measurement posture score that matches the body measurement target. It can be understood that in addition to calculating the body measurement posture score based on the elbows, waist, and legs, the target body measurement prediction model can also calculate the body measurement posture score based on the first 2D key points matched by other parts on the posture estimation heat map. This application does not limit the specific type and number of parts used for calculation.

[0094] In addition, it should be noted that the specific calculation process of the Euclidean distance is an existing technology, so it will not be repeated here. At the same time, the specific numerical value of the target preset posture score matched to each actual Euclidean distance can be set by the technician according to actual needs, and this application does not impose any restrictions on this.

[0095] In this way, the electronic device can identify the actual physical test posture adopted by the physical test target in the physical test item, thereby determining the physical test posture score corresponding to the actual physical test posture, and then generate a more comprehensive physical test evaluation result based on the physical test posture score.

[0096] In a feasible implementation manner, the step of "determining the target preset posture score that matches each of the actual Euclidean distances" in the above step S242 may specifically include steps S244 to S246:

[0097] Step S244: obtaining a plurality of preset Euclidean distances and preset posture scores corresponding to the plurality of preset Euclidean distances;

[0098] Step S245: screening the plurality of preset Euclidean distances based on the actual Euclidean distances to determine a target preset Euclidean distance that matches each of the actual Euclidean distances;

[0099] Step S246: Determine the target preset posture score that matches each actual Euclidean distance according to the preset posture score corresponding to each target preset Euclidean distance.

[0100] In this embodiment, after determining each actual Euclidean distance, the target body measurement prediction model may first read the above-mentioned storage module to obtain multiple preset Euclidean distances and preset posture scores that match each of the multiple preset Euclidean distances. Then, the target body measurement prediction model screens the multiple preset Euclidean distances based on the actual Euclidean distances to determine the target preset Euclidean distances that match each of the multiple actual Euclidean distances. Finally, the target body measurement prediction model determines the preset posture scores corresponding to each of the multiple target preset Euclidean distances as the target preset posture scores that match each of the multiple actual Euclidean distances.

[0101] Exemplarily, for example, after determining the actual Euclidean distances between the first 2D key points corresponding to the elbows, waist, legs and other parts and the matching second 2D key points, the target body measurement prediction model further reads the above-mentioned storage module to obtain a scoring mapping table containing multiple preset Euclidean distances and preset posture scores that match the multiple preset Euclidean distances. Afterwards, the target body measurement prediction model queries the scoring mapping table based on each actual Euclidean distance, thereby comparing each actual Euclidean distance with the multiple preset Euclidean distances contained in the scoring mapping table, so as to determine the target preset Euclidean distance that matches each actual Euclidean distance in the scoring mapping table. Finally, the target body measurement prediction model determines the preset posture scores that match each target preset Euclidean distance in the scoring mapping table as the target preset posture scores corresponding to the elbows, waist, legs and other parts.

[0102] It should be noted that the mapping relationship between each preset Euclidean distance and each preset posture score in the score mapping table can be set by the technician according to the needs, and this application does not impose any restrictions on this.

[0103] In this way, the electronic device can quickly identify the physical test posture score corresponding to the actual physical test posture adopted by the physical test target in the physical test item through a preset score mapping table, and then generate a more comprehensive physical test evaluation result according to the physical test posture score.

[0104] Step S30: generating fusion feature data based on the target image data, the body measurement posture score and the initial text data;

[0105] Step S40: determining a target body measurement evaluation result according to the fused feature data;

[0106] In this embodiment, after obtaining the body measurement posture score, the target body measurement prediction model calls its own configured encoder to process the target image data, the body measurement posture score and the acquired initial text data to obtain an image vector and a text vector of fixed dimensions, and maps the image vector and the text vector to the feature space, thereby obtaining fused feature data that contains both the semantic information of the image data and the semantic information of the text data. Finally, the target body measurement prediction model extracts the image information and text information contained in the fused feature data, and generates a target body measurement evaluation result based on the image information and the text information.

[0107] Exemplarily, for example, after obtaining the physical test posture score, the target physical test prediction model further calls multiple encoders configured by itself to process the posture evaluation heat map to obtain an image vector. At the same time, the target physical test prediction model calls multiple encoders to convert the physical test posture score, basic physical fitness text data, and cardiopulmonary function test text data to obtain a text vector, and then maps the image vector and the text vector to a shared feature space to obtain fused feature data that contains both the semantic information of the image data and the semantic information of the text data. Finally, the target physical test prediction model extracts the image information and text information contained in the fused feature data, processes the image information and text information based on visual comprehension ability and language comprehension ability, thereby generating an image-text joint representation, and generates a target physical test evaluation result based on the image-text joint representation.

[0108] It should be noted that the target physical test evaluation result may specifically include the number of completions of the physical test items performed by the physical test target, and / or whether the physical test target has passed the corresponding physical test items. It is understandable that the present application does not limit the specific content of the target physical test evaluation result.

[0109] In addition, in this embodiment and another embodiment, the target body measurement prediction model can also identify whether the body measurement posture adopted by the body measurement target is standard based on the semantic information corresponding to the body measurement posture score contained in the fusion feature data, and determine the posture improvement content based on the actual body measurement posture and the above-mentioned matching standard body measurement posture, and then add the posture improvement content to the target body measurement evaluation result, so that the body measurement target can improve its own body measurement posture movements based on the target body measurement evaluation result.

[0110] In this way, the target physical measurement prediction model can map image data, scoring data, text data, etc. to a shared feature space for fusion, so that the fused physical measurement data not only includes the dynamic motion information and posture information of the physical measurement target, but also includes the static body information of the physical measurement target, and then generates a more comprehensive physical measurement evaluation result based on the dynamic motion information, posture information and static body information.

[0111] In a feasible implementation manner, the above step S30 may specifically include steps S310 to S330:

[0112] Step S310: performing conversion processing based on the target image data to obtain a target image vector;

[0113] Step S320: performing conversion processing based on the body measurement posture score and the initial text data to obtain a target text vector.

[0114] Step S330: Mapping the target image vector and the target text vector to a target feature space to fuse them and generate fused feature data.

[0115] In this embodiment, after obtaining the body measurement posture score, the target body measurement prediction model first calls the image encoder configured by itself to convert the fused image data to obtain an image vector. Then, the target body measurement prediction model calls the text encoder configured by itself to convert the body measurement posture score and the above-mentioned initial text data to obtain a text vector. Finally, the target body measurement prediction model maps the generated image vector and text vector to a shared feature space to fuse them to obtain fused feature data containing the semantic information of the image data, the semantic information of the body measurement posture score, and the semantic information of the text data.

[0116] Exemplarily, for example, after obtaining the physical test posture score, the target physical test prediction model first calls the image encoder configured by itself to convert the posture estimation heat map, thereby converting the posture estimation heat map into an image vector of a fixed dimension. After that, the target physical test prediction model calls the text encoder configured by itself to convert the physical test posture score, basic physical fitness text data, and cardiopulmonary function test text data, thereby obtaining multiple fixed-dimensional text vectors. After that, the target physical test prediction model maps the image vector and each text vector to a shared feature space for fusion, thereby obtaining fused feature data that simultaneously contains the semantic information of the image data, the semantic information of the physical test posture score, the semantic information of the basic physical fitness text data, and the semantic information of cardiopulmonary function and other information.

[0117] In this embodiment, when the electronic device is running, it first calls a plurality of camera devices with different shooting angles configured by itself to shoot the physical measurement target, so as to capture a plurality of initial image data including the physical measurement target. At the same time, the electronic device reads the storage module configured by itself to obtain the initial text data pre-collected by the physical measurement target before the physical measurement. After that, the electronic device inputs the obtained multiple initial image data and initial text data into the target physical measurement prediction model configured by itself. The target physical measurement prediction model processes the multiple initial image data so that the semantics contained in the multiple initial image data affect each other, thereby fusing the target image data. The target physical measurement prediction model then processes the target The image data is identified to determine the actual body measurement posture adopted by the body measurement target during the body measurement process, and to determine the body measurement posture score corresponding to the actual body measurement posture. After that, the target body measurement prediction model calls its own configured encoder to process the target image data, the body measurement posture score and the acquired initial text data to obtain an image vector and a text vector of fixed dimensions, and maps the image vector and the text vector to the feature space, thereby obtaining fused feature data that contains both the semantic information of the image data and the semantic information of the text data. Finally, the target body measurement prediction model extracts the image information and text information contained in the fused feature data, and generates the target body measurement evaluation result based on the image information and the text information.

[0118] In this way, the present application solves the technical problem of low accuracy of physical fitness assessment results generated by electronic devices in the related art. That is, the present application obtains initial image data from multiple angles by shooting, thereby avoiding the situation where a single camera device is easily affected by ambient light during the shooting process. At the same time, the present application obtains target image data by fusing multiple initial image data, and obtains fused feature data by fusing the target image data with the posture score and text data, so as to generate physical fitness assessment results according to the fused feature data. In this way, the electronic device considers factors such as the dynamic motion performance and static body performance of the physical fitness target while generating the physical fitness assessment results, thereby achieving the technical effect of enabling the electronic device to generate more accurate physical fitness assessment results.

[0119] Based on the first embodiment of the present application, a second embodiment of the present application is proposed. In the second embodiment of the present application, the same or similar contents as those of the above embodiments can be referred to the above description and will not be described in detail later. Figure 3 , Figure 3 A flow chart of the second embodiment of the multimodal information fusion method for physical assessment provided in this application is shown in FIG. Figure 3 As shown, before the above step S10, the physical measurement and evaluation method of multimodal information fusion of the present application may further include steps A10 to A30:

[0120] Step A10: Obtain multiple preset positive sample pairs and negative sample pairs, wherein the positive sample pairs are sample pairs consisting of preset image data and matching preset text data, and the negative sample pairs are sample pairs consisting of preset image data and random preset text data.

[0121] Step A20: determining each distance difference between each of the positive sample pairs and each of the negative sample pairs, and determining a contrast loss function according to each of the distance differences.

[0122] Step A30: Based on the contrast loss function, the preset initial body measurement prediction model is adjusted to generate a target body measurement prediction model, so that the target body measurement prediction model processes the multiple initial image data and the initial text data to obtain a target body measurement evaluation result.

[0123] In this embodiment, before acquiring the initial image data and the initial text data, the electronic device may also first read the above-mentioned storage module to obtain multiple positive sample pairs consisting of preset image data and matching preset text data, and multiple negative sample pairs consisting of preset image data and random preset text data, and input each positive sample pair and each negative sample pair into the model training module configured by itself. After that, the model training module inputs each positive sample pair and each negative sample into a preset physical measurement prediction model, and the physical measurement prediction model processes each positive sample pair and each negative sample pair to learn the relationship between the image and the text and generate the distance difference between each positive sample pair and each negative sample pair. The model training module obtains the contrast loss function according to each distance difference. Finally, the model training module adjusts the model parameters of the initial physical measurement prediction model according to the contrast loss function to obtain the target physical measurement prediction model, so that the target physical measurement prediction model processes the multiple initial image data and initial text data corresponding to the physical measurement target to obtain the above-mentioned target physical measurement prediction result.

[0124] Exemplarily, for example, before the electronic device detects the physical measurement target to obtain multiple color RGB images and each initial text data, it can also read the storage module to obtain multiple positive sample pairs consisting of preset RGB images and matching text data, and obtain multiple negative sample pairs consisting of preset RGB images and random text data. The electronic device inputs each positive sample pair and each negative sample pair into the model training module configured by itself. After that, the model training module inputs each positive sample pair and each negative sample pair into the preset initial physical measurement prediction model, and the physical measurement prediction model trains each positive sample pair and each negative sample pair, thereby learning the image data. and the relationship between the text data, and determine the distance difference between each positive sample pair and each negative sample pair, the model training module then determines the contrast loss function based on each distance difference, and finally, the model training module adjusts the model parameters of the initial body measurement prediction model according to the contrast loss function to minimize the distance difference between each positive sample pair and each negative sample pair, thereby obtaining a target body measurement prediction model, and then the electronic device inputs the acquired color RGB image and each text data into the target body measurement prediction model, so that the body measurement prediction model outputs a target body measurement prediction result that matches the body measurement target based on the color RGB image and each text data.

[0125] It should be noted that the specific method of determining the contrast loss function based on the distance difference in the model training module is a prior art, so it will not be repeated here. In addition, the specific process of adjusting the model parameters based on the contrast loss function in the model training module is a prior art, so it will not be repeated here. In addition, it can be understood that the specific values ​​of each model parameter set in the initial body measurement estimation model can be modified by the technician according to actual needs, and this application does not limit this.

[0126] In this way, the electronic device can input multiple positive sample pairs and negative sample pairs into the physical measurement prediction model, so that the physical measurement prediction model can learn the relationship between image data and text data based on each sample pair, and then generate a target physical measurement prediction model that can process image data and text data corresponding to the physical measurement target.

[0127] This application also provides a multi-modal information fusion physical test evaluation device, please refer to Figure 4 , the multimodal information fusion physical measurement and evaluation device comprises:

[0128] The data acquisition module 10 is used to detect the body measurement target to obtain a plurality of initial image data, and to obtain initial text data corresponding to the body measurement target;

[0129] An image fusion module 20, configured to fuse a plurality of the initial image data to obtain target image data, and determine a body measurement posture score corresponding to the body measurement target according to the target image data;

[0130] A feature fusion module 30, configured to generate fused feature data based on the target image data, the body measurement posture score and the initial text data;

[0131] The result evaluation module 40 is used to determine the target body measurement evaluation result according to the fused feature data.

[0132] In a feasible implementation manner, the image fusion module 20 is further used for:

[0133] Extracting image features corresponding to each of the plurality of initial image data, and generating a plurality of heat maps corresponding to each of the initial image data according to each of the image features;

[0134] The first key points contained in each of the heat maps are determined, and the heat maps are fused based on the first key points to obtain target image data.

[0135] In a feasible implementation manner, the image fusion module 20 is further used for:

[0136] Determining a confidence level of each of the first key points for matching, and determining a fusion weight of each of the first key points for matching according to each of the confidence levels;

[0137] The heat maps are fused based on the fusion weights to obtain target image data.

[0138] In a feasible implementation manner, the image fusion module 20 is further used for:

[0139] Identify the actual body measurement posture corresponding to the body measurement target according to the target image data, and determine a standard body measurement posture matching the actual body measurement posture;

[0140] The second key points included in the standard body measurement posture are obtained, and a body measurement posture score corresponding to the body measurement target is determined according to the first key points and the second key points.

[0141] In a feasible implementation manner, the image fusion module 20 is further used for:

[0142] Determine the actual Euclidean distance between each of the first key points and the matched second key point;

[0143] Determine a target preset posture score that matches each of the actual Euclidean distances, and determine a preset score weight that corresponds to each of the first key points;

[0144] The preset posture scores are processed based on the preset scoring weights to determine the physical measurement posture score corresponding to the physical measurement target.

[0145] In a feasible implementation manner, the image fusion module 20 is further used for:

[0146] Acquire a plurality of preset Euclidean distances and preset posture scores corresponding to each of the plurality of preset Euclidean distances;

[0147] Screening the plurality of preset Euclidean distances based on the actual Euclidean distances to determine a target preset Euclidean distance that matches each of the actual Euclidean distances;

[0148] According to the preset posture scores corresponding to the target preset Euclidean distances, the target preset posture scores that match the actual Euclidean distances are determined.

[0149] In a feasible implementation manner, the above-mentioned result evaluation module 30 is further used for:

[0150] Performing conversion processing based on the target image data to obtain a target image vector;

[0151] Performing conversion processing based on the body measurement posture score and the initial text data to obtain a target text vector;

[0152] The target image vector and the target text vector are mapped to a target feature space to fuse and generate fused feature data.

[0153] In a feasible implementation manner, the data acquisition module 10 is further used for:

[0154] Acquire a plurality of preset positive sample pairs and negative sample pairs, wherein the positive sample pairs are sample pairs consisting of preset image data and matching preset text data, and the negative sample pairs are sample pairs consisting of preset image data and random preset text data;

[0155] Determine each distance difference generated between each of the positive sample pairs and each of the negative sample pairs, and determine a contrast loss function according to each of the distance differences;

[0156] Based on the contrast loss function, the preset initial body measurement prediction model is adjusted to generate a target body measurement prediction model, so that the target body measurement prediction model processes the multiple initial image data and the initial text data to obtain a target body measurement evaluation result.

[0157] The physical test and assessment device with multimodal information fusion provided by the present application adopts the physical test and assessment method with multimodal information fusion in the above-mentioned embodiment, which can solve the technical problem of low accuracy of physical test and assessment results generated by electronic devices in related technologies. Compared with the prior art, the beneficial effects of the physical test and assessment device with multimodal information fusion provided by the present application are the same as the beneficial effects of the physical test and assessment method with multimodal information fusion provided by the above-mentioned embodiment, and the other technical features in the physical test and assessment device with multimodal information fusion are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0158] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the physical measurement and assessment method of multimodal information fusion in the above-mentioned embodiment 1.

[0159] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, an electronic device with a plurality of camera devices with different angles and a model training module configured therein, or a mobile terminal, a data storage control terminal, a PC or other terminal connected to an electronic control unit supporting the electronic device. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0160] like Figure 5As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0161] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0162] The electronic device provided by the present application adopts the physical test and assessment method of multimodal information fusion in the above-mentioned embodiment, which can solve the technical problem of low accuracy of physical test and assessment results generated by electronic devices in related technologies. Compared with the prior art, the beneficial effects of the electronic device provided by the present application are the same as the beneficial effects of the physical test and assessment method of multimodal information fusion provided in the above-mentioned embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0163] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0164] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0165] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the multimodal information fusion physical measurement and assessment method in the above-mentioned embodiment.

[0166] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0167] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0168] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: detects a physical measurement target to obtain multiple initial image data, and obtains initial text data corresponding to the physical measurement target; performs fusion processing on the multiple initial image data to obtain target image data, and determines a physical measurement posture score corresponding to the physical measurement target based on the target image data; generates fused feature data based on the target image data, the physical measurement posture score and the initial text data; and determines a target physical measurement evaluation result based on the fused feature data.

[0169] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0170] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0171] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0172] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned multimodal information fusion physical measurement and evaluation method, and can solve the technical problem of low accuracy of physical measurement and evaluation results generated by electronic devices in related technologies. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the multimodal information fusion physical measurement and evaluation method provided in the above-mentioned embodiments, and will not be repeated here.

[0173] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned multi-modal information fusion physical measurement and assessment method.

[0174] The computer program product provided by the present application can solve the technical problem of low accuracy of the physical test evaluation results generated by electronic devices in the related art. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the physical test evaluation method of multimodal information fusion provided by the above embodiment, which will not be elaborated here.

[0175] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A multi-modal information fusion physical test assessment method, characterized in that: The multimodal information fusion physical measurement and evaluation method comprises: Detecting a body measurement target to obtain a plurality of initial image data, and obtaining initial text data corresponding to the body measurement target; Performing fusion processing on the plurality of the initial image data to obtain target image data, and determining a physical measurement posture score corresponding to the physical measurement target according to the target image data; generating fused feature data based on the target image data, the body measurement posture score and the initial text data; The target body measurement evaluation result is determined according to the fused feature data.

2. The multimodal information fusion physical assessment method according to claim 1, characterized in that: The step of fusing the plurality of initial image data to obtain target image data comprises: Extracting image features corresponding to each of the plurality of initial image data, and generating a plurality of heat maps corresponding to each of the initial image data according to each of the image features; The first key points contained in each of the heat maps are determined, and the heat maps are fused based on the first key points to obtain target image data.

3. The multimodal information fusion physical measurement and evaluation method according to claim 2, characterized in that: The step of fusing the heat maps based on the first key points to obtain target image data includes: Determining a confidence level of each of the first key points for matching, and determining a fusion weight of each of the first key points for matching according to each of the confidence levels; The heat maps are fused based on the fusion weights to obtain target image data.

4. The multimodal information fusion physical assessment method according to claim 2, characterized in that: The step of determining the physical measurement posture score corresponding to the physical measurement target according to the target image data comprises: Identify the actual body measurement posture corresponding to the body measurement target according to the target image data, and determine a standard body measurement posture matching the actual body measurement posture; The second key points included in the standard body measurement posture are obtained, and a body measurement posture score corresponding to the body measurement target is determined according to the first key points and the second key points.

5. The multimodal information fusion body measurement and evaluation method according to claim 4, characterized in that: The step of determining the physical test posture score corresponding to the physical test target according to each of the first key points and each of the second key points comprises: Determine the actual Euclidean distance between each of the first key points and the matched second key point; Determine a target preset posture score that matches each of the actual Euclidean distances, and determine a preset score weight that corresponds to each of the first key points; The preset posture scores are processed based on the preset scoring weights to determine the physical measurement posture score corresponding to the physical measurement target.

6. The multimodal information fusion physical assessment method according to claim 5, characterized in that: The step of determining the target preset posture score that matches each of the actual Euclidean distances comprises: Acquire a plurality of preset Euclidean distances and preset posture scores corresponding to each of the plurality of preset Euclidean distances; Screening the plurality of preset Euclidean distances based on the actual Euclidean distances to determine a target preset Euclidean distance that matches each of the actual Euclidean distances; According to the preset posture scores corresponding to the target preset Euclidean distances, the target preset posture scores that match the actual Euclidean distances are determined.

7. The multimodal information fusion physical assessment method according to claim 1, characterized in that: The step of generating fused feature data based on the target image data, the body measurement posture score and the initial text data comprises: Performing conversion processing based on the target image data to obtain a target image vector; Performing conversion processing based on the body measurement posture score and the initial text data to obtain a target text vector; The target image vector and the target text vector are mapped to a target feature space to fuse and generate fused feature data.

8. The multimodal information fusion physical measurement and evaluation method according to any one of claims 1 to 7, characterized in that: Before the step of detecting the body measurement target to obtain a plurality of initial image data, the method further includes: Acquire a plurality of preset positive sample pairs and negative sample pairs, wherein the positive sample pairs are sample pairs consisting of preset image data and matching preset text data, and the negative sample pairs are sample pairs consisting of preset image data and random preset text data; Determine each distance difference generated between each of the positive sample pairs and each of the negative sample pairs, and determine a contrast loss function according to each of the distance differences; Based on the contrast loss function, the preset initial body measurement prediction model is adjusted to generate a target body measurement prediction model, so that the target body measurement prediction model processes the multiple initial image data and the initial text data to obtain a target body measurement evaluation result.

9. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the multimodal information fusion physical measurement and assessment method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the multimodal information fusion physical measurement and evaluation method according to any one of claims 1 to 8 are implemented.

Images