Multi-modal AI algorithm and identification system applied to post evaluation in power industry

Abstract

The invention relates to a multi-modal AI algorithm and identification system applied to post evaluation in the power industry, and belongs to the technical field of electric digital data processing, and the method comprises the following steps: 1), training an SAM model and an ASR model which meet the requirements of an examination question scene; step 2): in the SAM model, constructing a visual component combined by a target identification type atomic component, a state identification type atomic component and a position change type atomic component; step 3), constructing a voice component in the ASR model; step 4), combining the visual component and the voice component to realize intelligent post assessment and evaluation; the system and the method have the beneficial effects that images or videos recognized by the SAM model and languages and characters recognized by the ASR model are assessed and evaluated through the assessment and evaluation module, so that the efficiency and the quality of intelligent and digital post assessment and evaluation are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of electrical digital data processing, and in particular relates to a multimodal AI algorithm and recognition system applied to job evaluation in the power industry. Background Art

[0002] The assessment and evaluation of professional skills and job competence plays a vital role in the power industry. It is not only the main yardstick for measuring the effectiveness of employee training and skill level, but also the indispensable core basis for the scientific selection and appointment of talents in the power industry. This system is not only related to the career growth of individual employees, but also directly related to the overall operational efficiency and service quality of power industry enterprises.

[0003] However, faced with the continuous emergence of new jobs and formats, as well as the rapid changes in the market environment, the existing job evaluation work still faces many challenges and urgently needs to be optimized and upgraded in the following key aspects:

[0004] (1) The traditional on-site job evaluation model has efficiency bottlenecks;

[0005] (2) The organization and coordination of traditional job evaluation business is cumbersome and complicated;

[0006] (3) In the traditional model, there are interactive risks in the direct interaction between candidates and examiners;

[0007] (4) Organizing job evaluation activities has a significant impact on grassroots work.

[0008] With the successful application of information and digital technologies such as big data and artificial intelligence in thousands of industries, power industry enterprises have also ushered in new opportunities for digital transformation. By introducing advanced technical means, not only can the efficiency and quality of job evaluation be effectively improved, but also the cost of manpower and material resources can be reduced, and the objectivity and fairness of evaluation can be enhanced. Therefore, power industry enterprises should actively explore the path of digital transformation, integrate information and digital technologies into job evaluation work, drive development with innovation, continuously improve the level of human resource management, and provide strong guarantees for the sustainable development of enterprises. Summary of the invention

[0009] The present invention provides a multimodal AI algorithm and recognition system for job assessment in the electric power industry, which is used to solve the technical problem of how to digitally improve the efficiency and quality of job assessment and evaluation in the prior art. The system combines visual components and voice components to conduct in-depth data analysis, that is, the system conducts in-depth mining and analysis of the candidate's operation data, and the system conducts in-depth mining and analysis of the candidate's voice, thereby improving the efficiency and quality of intelligent and digital job assessment and evaluation.

[0010] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0011] A multimodal AI algorithm applied to job evaluation in the power industry includes the following steps:

[0012] Step 1): Train the SAM model and ASR model that meet the requirements of the test scenario; the SAM model is used for image or video recognition, and the ASR model is used to convert the voice expressed by the examinee into language and text recognition;

[0013] Step 2): In the SAM model, construct a visual component composed of target recognition atomic components, state recognition atomic components and position change atomic components;

[0014] Step 3): In the ASR model, construct a speech component;

[0015] Step 4): Combine visual components and voice components to achieve intelligent job assessment and evaluation.

[0016] Optionally, in step 2), the score of the visual component is recorded in the test question, wherein the score of the visual component in the test question is recorded as .

[0017] Optionally, in step 2), the target recognition atomic component recognizes the spatial position relationship between two targets in the image or video, specifically using the following steps:

[0018] Step 01): Two spatial position relationship thresholds are set for each of the two targets. and ;

[0019] Step 02): Calculate the predicted box of an object And the predicted box of another target The intersection ratio ; The calculation method of the intersection-and-union ratio is formula (1): (1);

[0020] Step 03): Calculate the depth value between two targets ; The depth value between two targets is calculated as follows: (2); In formula (2), is the depth value that the SAM model can predict, is the maximum depth value between the two targets, used for normalization; is a scaling factor used to normalize the maximum depth value Restore to original size;

[0021] Step 04): According to the intersection ratio The two spatial position relationship thresholds of each target , Comparison, based on the depth value between the two targets The two-space position relationship threshold with each target , To judge the recognition results of the target recognition class atomic components by comparing .

[0022] Further, in step 04), the recognition result of the target recognition class atomic component , using the following formula (3):

[0023] (3);

[0024] In formula (3), if and , the prediction box of the target and the predicted box of the other target The intersection and union ratio Greater than the spatial position relationship threshold , the depth value between the two targets Less than the spatial position relationship threshold , the recognition is successful and is 1, otherwise the recognition fails and is 0.

[0025] Optionally, in step 2), the state recognition atomic component recognition is to recognize two labels for a target at different time points. The specific steps are as follows:

[0026] Step a): Take a target into two states and The two tags are identified respectively, and the identification result sequences of the two tags are and ;

[0027] Step b): Time has come At this moment, the recognition results of the two tags, 0 or 1, are appended to the The resulting sequence is as described In the result sequence;

[0028] Step c): Moment, yes The resulting sequence and The two sequences of the result sequence are XORed, and the result of the XORed is converted into decimal data for comparison; wherein, the result of the XORed is converted into decimal data for comparison using formula (4): (4); In formula (4), To perform XOR operation, To obtain the XOR result and convert it into decimal data, To find the XOR result, convert it into decimal data for comparison; if If the value is 1, the recognition is successful. If the value is 0, recognition fails.

[0029] Optionally, in step 2), the position change atomic component identification is to judge the spatiotemporal relationship of a target at different times, and the specific steps are as follows:

[0030] Step A): First set a target position change distance threshold and position change angle threshold ;

[0031] Step B): Time has come At the moment, find the position of a target in the field of view in space, and the positions in the field of view are The coordinate point of the prediction box of a target at the moment ,as well as The coordinate point of the prediction box of a target at the moment ;

[0032] Step C): Calculate the Euclidean distance between two coordinate points , specifically as follows formula (5):

[0033] (5);

[0034] Step D): Select The axis is horizontal, calculate the angle between two coordinate points , specifically as follows formula (6):

[0035] (6);

[0036] Step E): Calculate the recognition results of the spatiotemporal relationship of a target at different times , which is the following formula (7):

[0037] (7);

[0038] in, and ,but is 1, the recognition is successful; otherwise, If it is 0, the recognition fails.

[0039] Optionally, in step 3), the score of the speech component is recorded in the test question, wherein the score of the speech component in the test question is recorded as .

[0040] Optionally, in step 4), the score of the test question is calculated using the following steps:

[0041] Step I): The score of the visual component is calculated as follows: the confidence of the image or video clip recognized by the SAM model , multiplied by the score of the visual component ;

[0042] Step II): The score of the speech component is calculated by the sum of the weights of the keywords in the language and text recognized by the ASR model. , multiplied by the score of the voice component ;in, For the identified Keywords, is the total number of keywords identified;

[0043] Step III): Final score of the test question The calculation method is as follows:

[0044] (8);

[0045] in, For the A visual test point, To identify the The confidence level of the image or video clip for each visual test point , For the The score distribution of visual components of visual test points , For an exam question A visual test point, for The score of the visual component of each visual test point;

[0046] For the Voice test points, For the The distribution of points for the speech components of each speech test point , For an exam question Voice test points, for The score of the phonetic component of each phonetic test point.

[0047] A recognition system for multimodal AI algorithms used in job evaluation in the power industry, including:

[0048] ‌SAM model, used for image or video recognition;

[0049] ASR model, used to convert the voice of the examinee into language recognition;

[0050] Assessment and evaluation module, used for job assessment and evaluation;

[0051] The assessment and evaluation module is connected to the SAM model and the ASR model. The recognition data of the image or video in the SAM model and the recognition data of the language and text in the ASR model are transmitted to the assessment and evaluation module, and the assessment and evaluation module assesses and evaluates the position.

[0052] Optionally, the SAM model includes: an object recognition atomic component module for identifying the spatial position relationship between two objects in an image or video, an identification atomic component module for identifying two label states of an object at different time points, and a position change atomic component module for identifying and judging the spatiotemporal relationship of an object at different times;

[0053] The ASR model includes: a speech component module for converting speech into language and text recognition;

[0054] The assessment and evaluation module is connected to the target recognition atomic component module, the atomic component module and the position change atomic component module, and the assessment and evaluation module is connected to the speech component module;

[0055] Through the assessment and evaluation module, the visual component composed of the target recognition atomic component module, the atomic component module and the position change atomic component module and the speech component of the speech component module are combined for job assessment and evaluation.

[0056] Beneficial effects of the present invention:

[0057] The present invention utilizes visual components to digitally record operation details and voice components to record voice clips, and the visual components and voice components are combined to conduct in-depth data analysis: the system conducts in-depth mining and analysis on the examinee's operation data, and the system conducts in-depth mining and analysis on the examinee's voice data, thereby improving the efficiency and quality of intelligent and digital job assessment and evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0059] Figure 1 It is a schematic diagram of the structure of the identification system of the present invention;

[0060] Figure 2 It is a work flow chart of the present invention;

[0061] Figure 3 A schematic diagram of a recognition method of a visual component of the present invention;

[0062] Figure 4 It is a schematic diagram of the recognition method of the voice component of the present invention. DETAILED DESCRIPTION

[0063] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0064] Embodiment 1;

[0065] like Figure 1 As shown, this embodiment provides a recognition system for a multimodal AI algorithm applied to job evaluation in the power industry, including:

[0066] ‌SAM model, used for image or video recognition;

[0067] ASR model, used to convert the voice of the examinee into language recognition;

[0068] Assessment and evaluation module, used for job assessment and evaluation;

[0069] The assessment and evaluation module is connected to the SAM model and the ASR model. The recognition data of the image or video in the SAM model and the recognition data of the language and text in the ASR model are transmitted to the assessment and evaluation module, and the assessment and evaluation module assesses and evaluates the position.

[0070] The SAM model includes: an object recognition atomic component module for identifying the spatial position relationship between two objects in an image or video, an identification atomic component module for identifying two label states of an object at different time points, and a position change atomic component module for identifying and judging the spatiotemporal relationship of an object at different times;

[0071] The ASR model includes: a speech component module for converting speech into language and text recognition;

[0072] The assessment and evaluation module is connected to the target recognition atomic component module, the atomic component module and the position change atomic component module, and the assessment and evaluation module is connected to the speech component module;

[0073] Through the assessment and evaluation module, the visual component composed of the target recognition atomic component module, the atomic component module and the position change atomic component module and the speech component of the speech component module are combined for job assessment and evaluation.

[0074] In this embodiment, the images or videos recognized by the SAM model and the language and text recognized by the ASR model are evaluated through an evaluation module.

[0075] Embodiment 2;

[0076] Based on Example 1, Figure 2 As shown, this embodiment provides a multimodal AI algorithm applied to job evaluation in the power industry, including the following steps:

[0077] Step 1): Train a SAM model and an ASR model that meet the requirements of the test scenario; the SAM model is used for image or video recognition, and the ASR model is used to convert the voice expressed by the examinee into language and text recognition;

[0078] Step 2): In the SAM model, a visual component composed of a target recognition atomic component, a state recognition atomic component and a position change atomic component is constructed;

[0079] Step 3): In the ASR model, construct a speech component;

[0080] Step 4): Combine visual components and voice components to achieve intelligent job assessment and evaluation.

[0081] In step 2), in the test questions, the scores of the visual components are recorded, wherein the scores of the visual components in the test questions are recorded as .

[0082] In step 2), the target recognition atomic component recognizes the spatial position relationship between two targets in the image or video (i.e., the spatial position relationship between objects, or the spatial position relationship between human joints (hands, feet, head, etc.) and objects), specifically using the following steps:

[0083] Step 01): Set two spatial position relationship thresholds for each of the two targets and ;

[0084] Step 02): Calculate the predicted box of an object And the predicted box of another target The intersection ratio ; The calculation method of the intersection-and-union ratio is formula (1): (1);

[0085] Step 03): Calculate the depth value between two targets ; The depth value between two targets is calculated as follows: (2); In formula (2), is the depth value that the SAM model can predict, is the maximum depth value between the two targets, used for normalization; is a scaling factor used to normalize the maximum depth value Restore to original size;

[0086] Step 04): According to the intersection ratio The two-space position relationship threshold with each target , Comparison, based on the depth value between the two targets The two-space position relationship threshold with each target , To judge the recognition results of the target recognition class atomic components by comparing .

[0087] Further, in step 04), the recognition result of the target recognition class atomic component , using the following formula (3):

[0088] (3);

[0089] In formula (3), if and , a predicted box of an object And the predicted box of another target The intersection ratio Greater than the spatial position relationship threshold , the depth value between two targets Less than the spatial position relationship threshold , the recognition is successful and is 1, otherwise the recognition fails and is 0.

[0090] In step 2), the state recognition atomic component recognition is to identify two labels for a target at different time points. The specific steps are as follows:

[0091] Step a): Take a target into two states and (Because a target has different states at different time points) two labels are identified respectively, and the recognition result sequences of the two labels are and ;

[0092] Step b): Time has come At this moment, the recognition results of the two tags, 0 or 1, are appended to The result sequence and In the result sequence;

[0093] Step c): Moment, yes The resulting sequence and The two sequences of the result sequence are XORed, and the result of the XORed is converted into decimal data for comparison; wherein, the result of the XORed is converted into decimal data for comparison using formula (4): (4); In formula (4), To perform XOR operation, To obtain the XOR result and convert it into decimal data, To find the XOR result, convert it into decimal data for comparison; if If the value is 1, the recognition is successful. If the value is 0, recognition fails.

[0094] In step 2), the position change atomic component recognition is to judge the spatiotemporal relationship of a target at different times. The specific steps are as follows:

[0095] Step A): First set a target position change distance threshold and position change angle threshold ;

[0096] Step B): Time has come At this moment, find the position of a target in the field of view in space. The positions in the field of view are The coordinate point of the prediction box of a target at the moment ,as well as The coordinate point of the prediction box of a target at the moment ;

[0097] Step C): Calculate the Euclidean distance between two coordinate points , specifically as follows formula (5):

[0098] (5);

[0099] Step D): Select The axis is horizontal, calculate the angle between two coordinate points , specifically as follows formula (6):

[0100] (6);

[0101] Step E): Calculate the recognition results of the spatiotemporal relationship of a target at different times , which is the following formula (7):

[0102] (7);

[0103] in, and ,but is 1, the recognition is successful; otherwise, If it is 0, the recognition fails.

[0104] In step 3), the score of the speech component is recorded in the test question, wherein the score of the speech component in the test question is recorded as .

[0105] In step 4), the score calculation of the test question adopts the following steps:

[0106] Step I): The score of the visual component is calculated as follows: the confidence of the image or video clip recognized by the SAM model , multiplied by the score of the visual component ;

[0107] Step II): The score of the speech component is calculated by the sum of the weights of the keywords in the language and text recognized by the ASR model. , multiplied by the score of the voice component ;in, For the identified Keywords, is the total number of keywords identified;

[0108] Step III): Final score of the test question The calculation method is as follows:

[0109] (8);

[0110] in, For the A visual test point, To identify the The confidence level of the image or video clip for each visual test point , For the The score distribution of visual components of visual test points , For an exam question A visual test point, for The score of the visual component of each visual test point;

[0111] For the Voice test points, For the The distribution of points for the speech components of each speech test point , For an exam question Voice test points, for The score of the phonetic component of each phonetic test point.

[0112] Embodiment 3;

[0113] Based on Example 2, Figure 3 As shown, the target recognition atomic component, the state recognition atomic component and the position change atomic component are an ordered or disordered combination of one or more atomic components, which can meet a specific assessment requirement and are stateless. Take the vision class (checking safety helmets) as an example.

[0114] 1. Disassemble the inspection of the safety helmet into three atomic components (i.e., target recognition atomic components, state recognition atomic components, and position change atomic components): hold the safety helmet with both hands, let one hand hold the safety helmet and the other hand point to the safety helmet certificate; or hold the safety helmet with one hand and squeeze the helmet support with the other hand.

[0115] 2. Drag the above three atomic components to the arrangement area and set the properties of labels and time sequence.

[0116] 3. Publish the compiled visual components to the component market for use in subsequent exam question process compilation.

[0117] This example is a visual helmet inspection. By recognizing the image or video of both hands, the visual component of the examinee is scored. ,‌ The confidence level of the image or video clip automatically recognized by the SAM model , and obtain the score of the visual component.

[0118] Embodiment 4;

[0119] Based on Example 2, Figure 4 As shown in the figure, take the speech category as an example: convert the expressed speech into language text. For example, for the test point of wearing a helmet properly, it is necessary to configure the text "the helmet should not slide down when lowering the head and should not be loose when raising the head" and the weights of the keywords "helmet", "not slide down", and "not loose".

[0120] This embodiment converts speech into text to obtain the score of the candidate's speech component. , the sum of the weights of the keywords in the language and text automatically recognized by the ASR model , and obtain the score of the voice component.

[0121] Embodiment 4;

[0122] Based on all the above embodiments, the present invention is a visual component that digitally records operation details: the system can accurately capture and identify each operation detail of the examinee, save the operation details in digital form, and form a complete and detailed operation record.

[0123] The voice component records voice fragments. The system can accurately capture and recognize the candidate's voice, convert the captured voice into language and text, save the language and text in digital form, and form a complete and detailed voice record.

[0124] The system combines visual components and voice components to conduct in-depth data analysis: the system conducts in-depth mining and analysis on the candidates' operation data, as well as the system conducts in-depth mining and analysis on the candidates' voice data, thus improving the efficiency and quality of intelligent and digital job assessment and evaluation.

[0125] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal AI algorithm applied to job evaluation in the power industry, characterized in that: The steps include: Step 1): Train the SAM model and ASR model that meet the requirements of the test scenario; the SAM model is used for image or video recognition, and the ASR model is used to convert the voice expressed by the examinee into language and text recognition; Step 2): In the SAM model, construct a visual component composed of target recognition atomic components, state recognition atomic components and position change atomic components; Step 3): In the ASR model, construct a speech component; Step 4): Combine visual components and voice components to achieve intelligent job assessment and evaluation.

2. According to claim 1, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 2), the score of the visual component is recorded in the test question, wherein the score of the visual component is recorded as .

3. According to claim 1, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 2), the target recognition atomic component recognizes the spatial position relationship between two targets in an image or video, specifically using the following steps: Step 01): Two spatial position relationship thresholds are set for each of the two targets. and ; Step 02): Calculate the predicted box of an object And the predicted box of another target The intersection ratio ; The calculation method of the intersection-and-union ratio is formula (1): (1); Step 03): Calculate the depth value between two targets ; The depth value between two targets is calculated as follows: (2); In formula (2), is the depth value that the SAM model can predict, is the maximum depth value between the two targets, used for normalization; is a scaling factor used to normalize the maximum depth value Restore to original size; Step 04): According to the intersection-and-union ratio The two spatial position relationship thresholds of each target , According to the comparison of the depth value between the two targets The two-space position relationship threshold with each target , To judge the recognition results of the target recognition class atomic components by comparing .

4. According to claim 3, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 04), the recognition result of the target recognition class atomic component , using the following formula (3): (3); In formula (3), if and , the prediction box of the target and the predicted box of the other target The intersection and union ratio Greater than the spatial position relationship threshold , the depth value between the two targets Less than the spatial position relationship threshold , the recognition is successful and is 1, otherwise the recognition fails and is 0.

5. According to claim 1, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 2), the state recognition atomic component recognition is to recognize two labels for a target at different time points. The specific steps are as follows: Step a): The one target adopts two states and The two tags are identified respectively, and the identification result sequences of the two tags are and ; Step b): Time has come At this moment, the recognition results of the two tags, 0 or 1, are appended to the The resulting sequence is as described In the result sequence; Step c): Moment, yes The resulting sequence and The two sequences of the result sequence are XORed, and the result of the XORed is converted into decimal data for comparison; wherein, the result of the XORed is converted into decimal data for comparison using formula (4): (4); In formula (4), To perform XOR operation, To obtain the XOR result and convert it into decimal data, To find the XOR result, convert it into decimal data for comparison; if If the value is 1, the recognition is successful. If the value is 0, recognition fails.

6. According to claim 1, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 2), the position change atomic component identification is to judge the spatiotemporal relationship of a target at different times, and the specific steps are as follows: Step A): First set a target position change distance threshold and position change angle threshold ; Step B): Time has come At the moment, find the position of a target in the field of view in space, and the positions in the field of view are The coordinate point of the prediction box of a target at the moment ,as well as The coordinate point of the prediction box of a target at the moment ; Step C): Calculate the Euclidean distance between two coordinate points , specifically as follows formula (5): (5); Step D): Select The axis is horizontal, calculate the angle between two coordinate points , specifically as follows formula (6): (6); Step E): Calculate the recognition results of the spatiotemporal relationship of a target at different times , which is the following formula (7): (7); in, and ,but is 1, the recognition is successful; otherwise, If it is 0, the recognition fails.

7. According to claim 1, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 3), the score of the speech component is recorded in the test question, wherein the score of the speech component is recorded as .

8. According to claim 1, a multimodal AI algorithm applied to job evaluation in the power industry is characterized in that: In step 4), the score calculation of the test question adopts the following steps: Step I): The score of the visual component is calculated as follows: the confidence of the image or video clip recognized by the SAM model , multiplied by the score of the visual component ; Step II): The score of the speech component is calculated by the sum of the weights of the keywords in the language and text recognized by the ASR model. , multiplied by the score of the voice component ;in, For the identified Keywords, is the total number of keywords identified; Step III): Final score of the test question The calculation method is as follows: (8)； in, For the A visual test point, To identify the The confidence level of the image or video clip for each visual test point , For the The score distribution of visual components of visual test points , For a test question A visual test point, for The score of the visual component of each visual test point; For the Voice test points, For the The distribution of points for the speech components of each speech test point , For a test question Voice test points, for The score of the phonetic component of each phonetic test point.

9. A recognition system for a multimodal AI algorithm applied to job evaluation in the electric power industry, used to execute a multimodal AI algorithm applied to job evaluation in the electric power industry according to any one of claims 1 to 8, characterized in that: include: ‌SAM model, used for image or video recognition; ASR model, used to convert the voice of the examinee into language recognition; Assessment and evaluation module, used for job assessment and evaluation; The assessment and evaluation module is connected to the SAM model and the ASR model. The recognition data of the image or video in the SAM model and the recognition data of the language and text in the ASR model are transmitted to the assessment and evaluation module, and the assessment and evaluation module assesses and evaluates the position.

10. The identification system of a multimodal AI algorithm applied to job evaluation in the power industry according to claim 9 is characterized in that: The SAM model includes: a target recognition atomic component module for identifying the spatial position relationship between two targets in an image or video, a recognition atomic component module for identifying two label states of a target at different time points, and a position change atomic component module for identifying and judging the spatiotemporal relationship of a target at different times; The ASR model includes: a speech component module for converting speech into language and text recognition; The assessment and evaluation module is connected to the target recognition class atomic component module, the atomic component module and the position change class atomic component module, and the assessment and evaluation module is connected to the voice class component module; Through the assessment and evaluation module, the visual component composed of the target recognition atomic component module, the atomic component module and the position change atomic component module and the speech component of the speech component module are combined for job assessment and evaluation.

Images