Method, apparatus, device and storage medium for information processing

Abstract

Embodiments of the present disclosure relate to a method, apparatus, device and storage medium for information processing. The method proposed herein comprises: providing, to a first model, video content associated with an object to generate an action description text of at least one action of the object; and providing, to a second model, the action description text and first prompt information to generate evaluation content about the object, the first prompt information describing at least an action evaluation criterion associated with an evaluation target, the evaluation content indicating an evaluation result of the object about the evaluation target. In this way, embodiments of the present disclosure improve the accuracy of the output result.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computers, and more particularly to methods, apparatus, devices and computer-readable storage media for information processing. Background Technology

[0002] With the increasing maturity of computer technology, more and more users are using it to assist in work and learning. Users can use modeling techniques to analyze video content and output motion assessments of objects based on their movements. However, the accuracy of motion assessments output by traditional machine learning models is relatively poor. Summary of the Invention

[0003] In a first aspect of this disclosure, an information processing method is provided. The method includes: providing a first model with video content associated with an object to generate motion description text for at least one action of the object; and providing a second model with the motion description text and first cue information to generate evaluation content about the object, wherein the first cue information at least describes motion evaluation criteria associated with an evaluation target, and the evaluation content indicates the object's evaluation result regarding the evaluation target.

[0004] In a second aspect of this disclosure, an apparatus for information processing is provided. The apparatus includes: a first generation module configured to provide video content associated with an object to a first model to generate motion description text for at least one action of the object; and a second generation module configured to provide the motion description text and first prompt information to a second model to generate evaluation content about the object, wherein the first prompt information at least describes motion evaluation criteria associated with an evaluation target, and the evaluation content indicates the evaluation result of the object regarding the evaluation target.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown.

[0010] Figure 2 A flowchart illustrating an example process of information processing according to some embodiments of this disclosure is shown;

[0011] Figure 3 A flowchart illustrating an example process of information processing according to some embodiments of this disclosure is shown;

[0012] Figure 4 A schematic diagram of an information processing model according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic structural block diagram of an example apparatus for information processing according to some embodiments of the present disclosure is shown; and

[0014] Figure 6 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0019] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0020] As mentioned above, users can use machine learning models to analyze video content and output motion assessments of objects based on their movements. However, the accuracy of motion assessments output by traditional machine learning models is relatively poor.

[0021] Embodiments of this disclosure propose an information processing scheme. According to this scheme, video content associated with an object is provided to a first model to generate motion description text for at least one action of the object; and the motion description text and first prompt information are provided to a second model to generate evaluation content about the object, wherein the first prompt information at least describes motion evaluation criteria associated with the evaluation target, and the evaluation content indicates the object's evaluation result regarding the evaluation target.

[0022] In this manner, embodiments of this disclosure use two models for action understanding and result evaluation, respectively. First, embodiments of this disclosure process the video using a first model to obtain action description text of objects in the video. Then, embodiments of this disclosure analyze the action description text using a second model to obtain evaluation content about the objects. Compared to traditional methods that directly use models to analyze videos, the proposed solution allows both models to undergo more professional training, thereby improving the robustness of the entire information processing solution and enhancing the accuracy of its output results. Furthermore, the second model combines the first cue information describing the action evaluation criteria associated with the evaluation target to analyze the action description text, making the evaluation content output by the second model more closely match the user's evaluation needs, thereby further improving the accuracy of the evaluation content about the objects.

[0023] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0024] Example Environment

[0025] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. For example... Figure 1 As shown, example environment 100 may include electronic device 110. In some embodiments, electronic device 110 may provide a first model with video content associated with an object to generate motion description text for at least one action of the object; and provide a second model with motion description text and first prompt information to generate evaluation content about the object, wherein the first prompt information at least describes the motion evaluation criteria associated with the evaluation target, and the evaluation content indicates the evaluation result of the object with respect to the evaluation target.

[0026] Model 120 can be deployed on electronic device 110, or on other devices, which will not be elaborated here. Electronic device 110 can output the above evaluation results through model 120. Model 120 may include a first model and a second model.

[0027] In some embodiments, the electronic device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, handheld computers, portable gaming terminals, VR / AR devices, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, the electronic device 110 may also support any type of interface for the target user (such as "wearable" circuitry).

[0028] Electronic device 110 can also be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Electronic device 110 may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0029] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0030] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0031] Example process

[0032] Figure 2 A flowchart of an example process 200 for information processing according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 110. Process 200 can be applied to motion assessment based on video motion recognition, such as disease diagnosis, scoring of sports such as gymnastics and dance, etc., without limitation.

[0033] like Figure 2 As shown in box 210, electronic device 110 provides video content to the first model to generate action description text for at least one action of the object.

[0034] Figure 3 A flowchart of an example process 300 for information processing according to some embodiments of the present disclosure is shown.

[0035] like Figure 3 As shown, in some embodiments, user 305 can actively capture or upload video content via electronic device 110. Electronic device 110 processes the video content through evaluation system 310 deployed thereon to generate evaluation content 320 about objects in the video content.

[0036] In other embodiments, the electronic device 110 may also connect to an external device 315, such as a home security camera. With the user's authorization, the electronic device 110 periodically or irregularly acquires video content about the user from the external device 315. The electronic device 110 then processes the video content through an evaluation system 310 deployed on it to generate evaluation content 320 about objects in the video content.

[0037] In some embodiments, the aforementioned object may refer to one or more objects in the video content. The video content associated with the object may be a video of the object shot using non-professional imaging equipment, such as a mobile phone, home video camera, or other everyday shooting equipment. The video content may include content related to the object's daily life, work, or activities, without limitation. Alternatively, the video content may be uploaded from a storage device such as a computer or server that stores video content. Actions may include everyday actions such as walking, raising a hand, or picking up and placing objects, or preset actions such as holding one's head with both hands, raising both arms horizontally, or circling both arms. The action description text may be text describing the object performing the aforementioned actions.

[0038] Figure 4 A schematic diagram of an information processing model 400 according to some embodiments of the present disclosure is shown.

[0039] like Figure 4 As shown, in some embodiments, the first model 410 is a multimodal model, that is, a machine learning model capable of simultaneously processing and understanding multiple types of data, such as text, images, audio, video, etc. For example, the first model 410 may be a visual language model.

[0040] The first model 410 can receive one or more video contents 402 associated with an object, understand and analyze the actions of the object in the video contents 402, and generate action description text 412 for the object in the video contents 402. For example, the video contents 402 may include content describing the object's movements. The first model 410 can generate corresponding action description text 412, such as the object raising both hands simultaneously. For example, the first model 410 can output action description text 412 corresponding to multiple actions of the object based on the video contents 402, such as action 1 raising both hands, action 2 holding hands behind the head, etc.

[0041] In some embodiments, the electronic device 110 further provides a second prompt information 404 to the first model 410, and then the first model 410 can generate action description text 412 of the objects in the video content 402 based on the second prompt information 404 and the video content 402. As an example, the second prompt information 404 may indicate an action recognition standard associated with a set of preset actions.

[0042] Taking a video-based diagnosis of shoulder joint disease as an example, the preset actions can be reference actions that reflect shoulder joint disease. For example, an abduction action, i.e., raising the arm to the sides of the body; or a back-touching action, i.e., the subject touches the opposite scapula with one hand from above and the back with the other hand from below. Through the second prompt information 404, the first model 410 can more efficiently and accurately identify whether the subject in the video has performed the above preset actions.

[0043] In some embodiments, the second prompt information 404 may also indicate a set of capabilities associated with the first model 410, through which the first model 410 can generate more detailed and accurate action description text 412.

[0044] In some embodiments, a set of capabilities includes a first capability for recognizing the actions of an object, used to capture the actions of an object in video content 402. The first model 410 can recognize that an object in the video content has performed actions such as action 1, action 2, action 3, and action 4, which may include preset actions.

[0045] In some embodiments, the set of capabilities further includes a second capability describing the trajectory of the corresponding action. As an example, action 1 is raising both arms. The first model 410 can specifically describe this arm-raising action using the second capability, for example, raising the left arm to the left side of the body to 180° and the right arm to the right side of the body to 150°.

[0046] In some embodiments, the set of capabilities further includes a third capability to analyze the symmetry of the corresponding actions. For example, when an object raises both arms, the left arm is raised to the left of the body to 180° and the right arm is raised to the right of the body to 150°. The first model 410 can analyze this using the third capability to determine that the angles at which the left and right arms are raised are asymmetrical when the object performs action 1. Accordingly, the action description text 412 can describe that the action has an asymmetrical angle at which the left and right arms are raised.

[0047] In some embodiments, the set of capabilities further includes a fourth capability to analyze compensation information for the corresponding action. The first model 410 can use the fourth capability to determine whether there is compensation during the object's completion of the action, based on the action's specifications and actual performance. For example, the specification for the action of raising both arms is that the shoulder joints remain in a depressed position while raising both arms. If the object shrugs its shoulders while raising its arms, the first model 410 can analyze this using the fourth capability and determine that the object exhibits shoulder joint compensation during action 1. Accordingly, the action description text 412 can describe that the action involves shoulder joint compensation.

[0048] In some embodiments, the set of capabilities further includes a fifth capability for analyzing the smoothness of the corresponding action. The first model 410 can use this fifth capability to determine the smoothness of the object's performance of the action based on factors such as whether the action is accompanied by trembling or significant pauses. As an example, the first model 410 can analyze whether the object's action is smooth. For instance, if the object's action has significant pauses, the action description text 412 can describe the object's smoothness in performing action 1 as low.

[0049] In this way, the first model 410 can generate action description text 412 that is more closely matched to the evaluation target using the aforementioned capabilities. Since action description itself is quite broad—for example, the action of raising both arms—its related description can also include the movements and postures of the elbows, wrists, and fingers on both sides when raising the arms. The first model 410 guides the direction and focus of the action description through the aforementioned second prompt information 404, thereby obtaining a more accurate and effective action description text 412.

[0050] In some embodiments, the second prompt information 404 can also be configured to instruct the first model 410 to generate motion description text 412 using relative position description. For example, in the diagnostic criteria for shoulder joint diseases, since each individual's basic data such as height, weight, and arm span differ, the positional relationship between the arm and parts such as the shoulder, back, and head can more accurately determine the range of motion of the shoulder joint compared to diagnosing based on the range of motion of the arm. For example, the motion description text 412 generated accordingly could be, for instance, the arm touching the opposite scapula backward, or the arm being raised overhead.

[0051] In this way, the second prompt information 404 combines dynamic action descriptions and static posture descriptions to more accurately describe the actions of objects in the video content 402, so that the first model 410 can generate action description text 412 that is more relevant to the evaluation target. In other embodiments, the second prompt information 404 can also be adjusted according to different evaluation targets, which is not limited here.

[0052] Continue to refer to Figure 2In box 220, electronic device 110 provides action description text 412 and first prompt information 414 to second model 420 to generate evaluation content 425 about the object. The first prompt information 414 at least describes the action evaluation criteria associated with the evaluation target, and the evaluation content 425 indicates the evaluation result of the object about the evaluation target.

[0053] In some embodiments, the assessment target can be a movement assessment based on action recognition, such as the diagnosis of spinal diseases or Parkinson's disease where symptoms can be judged through limb movements; or the scoring of standard gymnastics, dance, and other sports events based on whether the movements meet preset standards. The following description uses a shoulder joint disease as an example. As an example, the first prompt information 414 describes the movement assessment criteria associated with shoulder joint disease, such as the related symptoms of shoulder joint disease. Accordingly, the assessment result of the object regarding the assessment target can be the judgment result of whether the object has shoulder joint disease.

[0054] Continue to refer to Figure 4 In some embodiments, the action description text 412 generated by the first model 410 can be used as input to the second model 420. The second model 420 is a reasoning model different from the first model 410. For example, the second model 420 can be a language model.

[0055] The first model 410 is configured as a multimodal model to facilitate the reception and understanding of different video content 402, thereby improving the accuracy of video content 402 recognition. The second model 420 is configured as an inference model focused on comparative analysis, thereby improving the accuracy of judging the recognized actions.

[0056] In some embodiments, the second model 420 generates the evaluation content of the object based on the first model 410 and the first prompt information. The first prompt information is generated by the third model 430. The third model 430 is a multimodal model different from the first model 410 and the second model 420. As an example, the third model 430 can be deployed in an external device that can communicate with the electronic device 110, and the external device sends the first prompt information determined by the third model 430 to the second model 420.

[0057] In some embodiments, the third model 430 can receive multiple reference contents 320 associated with the assessment target, generate standard summary information based on the multiple reference contents 320, and then the third model 430 can determine the first prompt information 414 based on the standard summary information. As an example, in the scenario of diagnosing shoulder joint diseases, the multiple reference contents 320 associated with the assessment target can be media content related to the diagnosis of shoulder joint diseases, such as diagnostic videos or popular science videos, audio recordings, etc.; or text content related to shoulder joint diseases, such as case studies, textbooks, papers, popular science articles, etc. As an example, several popular science videos about shoulder joint diseases mention that patients with shoulder joint diseases perform a back-touching action, i.e., touching the opposite scapula with one hand from above and the back with the other hand from below. If the hands cannot touch each other, the shoulder joint movement is restricted. The corresponding standard summary information can be the action assessment standard for the back-touching action, i.e., touching the opposite scapula with one hand from above and the back with the other hand from below, with the hands needing to touch each other. In this way, the third model 430 can extract standard summary information from actual diagnostic cases and disease-related professional data, thereby improving the professionalism and accuracy of the first prompt information.

[0058] In some embodiments, the electronic device 110 can employ cue word engineering strategies for the first and second cue messages, such as the chain of thought (COT) strategy and the self-consistency (SC) strategy. The COT strategy is a reasoning enhancement technique that guides the model to generate answers step-by-step by explicitly decomposing the task into multiple logical steps. The SC strategy is a quality optimization technique that selects the result with the highest consistency or optimal coverage by generating the same task multiple times. Combining these two strategies forms an efficient generative optimization framework, improving the accuracy of the generated evaluation results.

[0059] In some embodiments, the motion evaluation criteria associated with the evaluation target can indicate execution standards for multiple preset actions. As an example, execution standards may include recognition standards, trajectory standards, and symmetry standards. Taking abduction as an example, the recognition standard could be raising the arms from both sides of the body, which allows the abduction action to be identified. The trajectory standard could be extending the arms to 120° to 180°, which allows the completion of the abduction action to be determined. The symmetry standard could be that the angles at which both arms are raised are substantially the same, for example, the difference in angle is less than 10°, which allows the completion of the abduction action for each arm to be determined separately.

[0060] In some embodiments, the second model 420 can generate assessment content about the object based on the action assessment criteria and action description text associated with the assessment target. For example, if the action description text involves three actions of the object, the three actions are assessed based on the first prompt information. Based on the action assessment results of whether the three actions meet the action assessment criteria, the assessment content of whether the object suffers from shoulder joint disease can be obtained.

[0061] In this way, the embodiments of this disclosure use two models, a first model 410 and a second model 420, to perform action understanding and result evaluation respectively, thereby improving the robustness of the entire information processing scheme and enhancing the accuracy of its output results.

[0062] In some embodiments, the electronic device 110 obtains an assessment result of the object through a first model 410 and a second model 420, i.e., whether the object suffers from shoulder joint disease. Based on this, the electronic device 110 can generate evaluation information associated with the assessment result by comparing the object's assessment result with a reference result. The reference result can be the object's actual diagnostic result, which can be obtained by a doctor's actual diagnosis or based on the object's medical records or medical diagnostic images. By comparing the reference result with the assessment result, the electronic device 110 can obtain evaluation information of the assessment result, i.e., whether the assessment result is accurate. For example, if the object's assessment result is the same as the reference result, the assessment result is correct; otherwise, the assessment result is incorrect.

[0063] In some embodiments, the electronic device 110 can also obtain the overall evaluation accuracy based on the evaluation information of the evaluation results of multiple objects. For example, if the electronic device 110 generates evaluation results for a total of 100 objects, of which 90 evaluation results are the same as their corresponding reference results and the other 10 evaluation results are different from their corresponding reference results, then the evaluation accuracy of the electronic device 110 is 90%.

[0064] In some embodiments, evaluation information may be determined not only based on the accuracy of the results. Evaluation information may also be determined based on specific diagnostic details, such as the completeness of identification associated with the motion description text and the reasonableness of the assessment associated with the motion assessment criteria.

[0065] In some embodiments, the identification integrity of the electronic device 110 can be determined based on the number of actions identified by the first model 410. The identification integrity judgment rules are shown in Table 1 below.

[0066] Table 1

[0067]

[0068] As an example, the object in the video content completes a total of 4 actions: Action 1, Action 2, Action 3, and Action 4. If the action description text describes that the object completed 4 actions (Action 1, Action 2, Action 3, and Action 4), then the first model 410 can capture all the actions of the object in the video, and the recognition completeness A is 1. If the action description text describes that the object completed 3 actions (Action 1, Action 2, and Action 4), then the first model 410 misses one action and can only capture part of the object's actions in the video, and the recognition completeness A is 0.5. If the action description text describes that the object completed 0 actions, then the first model 410 misses all of the object's actions, and the recognition completeness A is 0.

[0069] In some embodiments, recognition completeness can provide feedback on whether the first model 410 can completely recognize the action performed by the object. The electronic device 110 can adjust the first model 410 based on this recognition completeness to improve its recognition accuracy. As an example, if the first model 410 consistently misses a specific action, such as action 3, in multiple video contents, the second prompt information regarding action 3 can be adjusted.

[0070] In some embodiments, the reasonableness of the evaluation of the electronic device 110 can be determined based on the action judgment result output by the second model 420. The judgment rules for the reasonableness of the evaluation are shown in Table 2 below.

[0071] Table 2

[0072]

[0073] As an example, the object in the video content completes four actions: Action 1, Action 2, Action 3, and Action 4, and all four actions meet their corresponding action evaluation criteria. If the second model 420 considers Action 1, Action 2, Action 3, and Action 4 to all meet their corresponding action evaluation criteria, then the second model 420's judgment on each action is consistent with the action evaluation criteria, and the evaluation rationality R = 1. If the second model 420 considers Action 1, Action 2, and Action 4 to meet their corresponding action evaluation criteria, but Action 3 does not, then the second model 420's judgment on Action 3 is a misjudgment, and the evaluation rationality R = 0.5. If the second model 420 considers Action 1, Action 2, Action 3, and Action 4 to not meet their corresponding action evaluation criteria, then the second model 420's judgment on each action contradicts the original video content, and the evaluation rationality R = 0.

[0074] In some embodiments, the assessment rationality can provide feedback on whether the second model 420 can accurately determine whether each action meets the action assessment criteria. The electronic device 110 can adjust the second model 420 based on the assessment rationality to improve its assessment accuracy. As an example, if the second model 420 consistently makes errors in its judgment of a specific action, such as action 3, the first prompt information regarding action 3 can be adjusted.

[0075] In some embodiments, a specific action among multiple reference actions, such as a preset action, can be set as a key action. The above-mentioned action count and action judgment accuracy statistics can be associated only with the key actions, thereby simplifying calculations and improving efficiency.

[0076] In some embodiments, the accuracy of the results from the electronic device 110 can be determined based on the evaluation results of the object and the reference results. The rules for judging the accuracy of the results are shown in Table 3 below.

[0077] Table 3

[0078]

[0079] As an example, the subject in the video content suffers from a shoulder joint disorder. If the assessment concludes that the subject has a shoulder joint disorder, the assessment result is consistent with the actual diagnosis, and the accuracy of the result is D=1. If the assessment result concludes that the subject does not have a shoulder joint disorder, the assessment result is inconsistent with the actual diagnosis, and the accuracy of the result is D=0.

[0080] In some embodiments, when judging the usability of the entire process 200, the electronic device 110 can make a comprehensive judgment by combining the identification completeness A, assessment rationality R, and result accuracy D. As an example, the weights of identification completeness A, assessment rationality R, and result accuracy D can be set to 0.2, 0.3, and 0.5, respectively. Therefore, the formula for calculating the usability coefficient UI is: UI = 0.5 * D + 0.3 * R + 0.2 * A. The usability of process 200 can be judged based on UI. For example, when UI is greater than a preset threshold, the current process 200 performs well; when UI is less than the preset threshold, the current process 200's performance needs improvement. It is worth mentioning that the identification completeness A, assessment rationality R, and result accuracy D, and their respective weights, are all illustrative examples. In practical applications, more diagnostic details and different evaluation criteria can be combined to generate the above usability evaluation information.

[0081] It should be understood that the specific weighting coefficients and scoring rules mentioned above are intended as examples and do not constitute a limitation of this disclosure.

[0082] In this manner, embodiments of this disclosure use two models for action understanding and result evaluation, respectively. First, embodiments of this disclosure process the video using a first model to obtain action description text of objects in the video. Then, embodiments of this disclosure analyze the action description text using a second model to obtain evaluation content about the objects. Compared to traditional methods that directly use models to analyze videos, the proposed solution allows both models to undergo more professional training, thereby improving the robustness of the entire information processing solution and enhancing the accuracy of its output results. Furthermore, the second model combines the first cue information describing the action evaluation criteria associated with the evaluation target to analyze the action description text, making the evaluation content output by the second model more closely match the user's evaluation needs, thereby further improving the accuracy of the evaluation content about the objects.

[0083] In some embodiments, the first, second, and / or third models can be fine-tuned using a dataset associated with the evaluation objective. As an example, low-rank adaptation of the large language model (LoRA) and supervised fine-tuning (SFT) can be used to make the first, second, and / or third models more closely match their respective analysis objectives. In this way, the evaluation content generated by the above scheme can better align with the evaluation objective, improving the accuracy of the evaluation content regarding the object.

[0084] Example devices and equipment

[0085] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an example apparatus 500 for information processing according to certain embodiments of the present disclosure is shown. Apparatus 500 may be implemented as or included in electronic device 110. Various modules / components in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0086] like Figure 5 As shown, the apparatus 500 includes: a first generation module 510 configured to provide video content associated with an object to a first model to generate motion description text for at least one action of the object; and a second generation module 520 configured to provide motion description text and first prompt information to a second model to generate evaluation content about the object, wherein the first prompt information at least describes the motion evaluation criteria associated with the evaluation target, and the evaluation content indicates the evaluation result of the object regarding the evaluation target.

[0087] In some embodiments, the first prompt information is generated based on the following process: providing a third model with multiple reference contents associated with the evaluation objective to obtain standard summary information generated by the third model based on the multiple reference contents; and determining the first prompt information based on the standard summary information.

[0088] In some embodiments, multiple reference contents include media content and / or text content associated with the evaluation objective.

[0089] In some embodiments, the first generation module 510 is further configured to provide the first model with second prompt information and video content to obtain action description text generated by the first model based on the second prompt information and video content, wherein the second prompt information indicates an action recognition standard associated with a set of preset actions.

[0090] In some embodiments, the second prompt information also indicates a set of capabilities associated with the first model, the set of capabilities including at least one of the following: a first capability for identifying the action of an object; a second capability for describing the trajectory of the corresponding action; a third capability for analyzing the symmetry of the corresponding action; a fourth capability for analyzing the compensatory information of the corresponding action; and a fifth capability for analyzing the smoothness of the corresponding action.

[0091] In some embodiments, the second prompt is configured to instruct the first model to use relative position description to generate action description text.

[0092] In some embodiments, the video content includes first video content that has been filmed; and / or second video content that has been uploaded.

[0093] In some embodiments, the apparatus 500 further includes a third generation module configured to generate evaluation information associated with the evaluation results, at least based on a comparison between the evaluation results of the object and reference results.

[0094] In some embodiments, the apparatus 500 further includes an adjustment module configured to adjust the first model and / or the second model using evaluation information.

[0095] In some embodiments, the evaluation information is further determined based on at least one of the following: identification completeness associated with the action description text, which is determined based on the number of actions identified by the first model; and evaluation reasonableness associated with the action evaluation criteria, which is determined based on the action judgment results output by the second model.

[0096] In some embodiments, the assessment content describes disease assessment information associated with a preset disease, which is associated with the object's behavioral representation.

[0097] The units included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0098] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 1 Electronic devices 110.

[0099] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0100] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 600.

[0101] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0102] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0103] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0104] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0105] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0106] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0107] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0109] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An information processing method, comprising: Provide the first model with video content associated with the object to generate motion description text for at least one action of the object; as well as The action description text and first prompt information are provided to the second model to generate evaluation content about the object. The first prompt information describes at least the action evaluation criteria associated with the evaluation target. The evaluation content indicates the evaluation result of the object with respect to the evaluation target.

2. The method according to claim 1, wherein the first prompt information is generated based on the following process: Provide the third model with multiple reference contents associated with the evaluation objective to obtain standard summary information generated by the third model based on the multiple reference contents; and Based on the summarized standard information, the first prompt information is determined.

3. The method of claim 2, wherein the plurality of reference contents includes media content and / or text content associated with the evaluation target.

4. The method of claim 1, wherein generating the action description text for at least one action of the object comprises: The first model is provided with a second prompt and the video content to obtain the action description text generated by the first model based on the second prompt and the video content. The second prompt indicates an action recognition standard associated with a set of preset actions.

5. The method of claim 4, wherein the second prompt information further indicates a set of capabilities associated with the first model, the set of capabilities including at least one of the following: The first capability is used to identify the actions of the object; The second ability is used to describe the trajectory of the corresponding action; The third ability is used to analyze the symmetry of corresponding actions; The fourth ability is used to analyze compensatory information for corresponding actions; The fifth ability is used to analyze the smoothness of corresponding movements.

6. The method of claim 4, wherein the second prompt information is configured to instruct the first model to generate the action description text using a relative position description.

7. The method according to claim 1, wherein the video content includes: The first video content that was filmed; and / or The content of the second video that has been uploaded.

8. The method according to claim 1, further comprising: Evaluation information associated with the evaluation results is generated, at least based on a comparison between the evaluation results of the object and reference results.

9. The method according to claim 8, further comprising: The evaluation information is used to adjust the first model and / or the second model.

10. The method of claim 8, wherein the evaluation information is further determined based on at least one of the following: The recognition completeness associated with the action description text, which is determined based on the number of actions recognized by the first model; The reasonableness of the assessment associated with the action assessment criteria is determined based on the action judgment results output by the second model.

11. The method of claim 1, wherein the assessment content describes disease assessment information associated with a preset disease, the preset disease being associated with the action representation of the object.

12. An apparatus for information processing, comprising: The first generation module is configured to provide video content associated with an object to the first model in order to generate motion description text for at least one action of the object; as well as The second generation module is configured to provide the action description text and the first prompt information to the second model to generate evaluation content about the object. The first prompt information describes at least the action evaluation criteria associated with the evaluation target, and the evaluation content indicates the evaluation result of the object with respect to the evaluation target.

13. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 11.

Images