Target identification method, apparatus and device, and computer storage medium

Abstract

The invention provides a target identification method, device and equipment and a computer storage medium, and the method relates to the technical field of computers, and is used for improving the efficiency and objectivity of assessment score evaluation in practical operation assessment. The method comprises the steps of obtaining at least two key frames meeting a preset pixel condition from a video to be identified based on pixel differences corresponding to pixel positions between adjacent frames in the video to be identified; based on the image features in the at least two key frames, performing target recognition processing on the at least two key frames to obtain corresponding target recognition results; the target recognition result represents a target operation action corresponding to a target object in the key frame and a target device state of at least one target device; and performing evaluation processing on the target operation and the at least one target equipment state based on the target operation action and a preset corresponding relationship between the at least one target equipment state and an evaluation standard, and obtaining an operation evaluation result corresponding to the target object.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device and equipment for target recognition and a computer storage medium. Background Art

[0002] With the continuous development of information technology, in examinations involving practical operations, video recording equipment has become increasingly popular in examination venues to record candidates' practical actions and results for subsequent review.

[0003] On the other hand, for those tests involving practical operations, the evaluation criteria are based on the traditional manual evaluation method. For example, in the physical, chemical and biological experiment test, when the test scores are measured, manual observation and evaluation of the test takers' experimental operations and experimental results are required.

[0004] This results in a high labor and time cost to obtain the corresponding assessment results in an assessment, and there will also be certain subjective factors that affect the assessment results of the candidates.

[0005] Therefore, how to further utilize the video equipment in the examination venue and improve the efficiency and objectivity of the assessment of examination results is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present application provide a method, apparatus, device and computer storage medium for target recognition, which are used to improve the efficiency and objectivity of assessment of assessment results in practical assessments.

[0007] In a first aspect, an embodiment of the present application provides a method for target recognition, comprising:

[0008] Acquire a video to be identified; the video to be identified is used to record the operation action of the target object on at least one target device, and the device status of the at least one target device;

[0009] Based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified, at least two key frames satisfying a preset pixel condition are obtained from the video to be identified; wherein the at least two key frames record: an action change in which the pixel difference corresponding to the target object exceeds a first change threshold and / or a state change in which the pixel difference corresponding to at least one target device exceeds a second change threshold;

[0010] Based on the image features in the at least two key frames, target recognition processing is performed on the at least two key frames to obtain corresponding target recognition results; the target recognition results represent: the target operation action corresponding to the target object in the key frames, and the target device state of the at least one target device;

[0011] Based on the preset corresponding relationship between the target operation action and at least one of the target device states and the evaluation criteria, the target operation and at least one of the target device states are evaluated and processed to obtain the operation evaluation result corresponding to the target object.

[0012] Optionally, the acquiring at least two key frames satisfying a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified includes:

[0013] For any two adjacent image frames in the video to be identified, perform the following operations respectively:

[0014] Based on the pixel values ​​corresponding to the two adjacent image frames, a difference operation is performed to obtain the pixel difference corresponding to each pixel position in the two adjacent image frames;

[0015] When the sum of the absolute values ​​of the pixel differences is greater than or equal to the first difference threshold included in the preset pixel condition, a key frame corresponding to the video to be identified is obtained based on the two adjacent image frames.

[0016] Optionally, acquiring a key frame corresponding to the to-be-identified video based on the two adjacent image frames includes:

[0017] Based on the numerical relationship between the pixel difference corresponding to each pixel position in the two adjacent image frames and the second difference threshold value included in the preset pixel condition, the pixels in the two adjacent image frames are divided into foreground pixels and background pixels;

[0018] For each of the two adjacent image frames, the following operations are performed respectively: the pixel value corresponding to the background pixel in one image frame is adjusted to a preset pixel value, and the image frame after the pixel value is adjusted is used as a key frame corresponding to the video to be identified.

[0019] Optionally, performing target recognition processing on the at least two key frames based on the image features in the at least two key frames to obtain corresponding target recognition results includes:

[0020] Using a trained target recognition model, based on image features in the at least two key frames, performing target recognition processing on the at least two key frames to obtain corresponding target recognition results;

[0021] The target recognition model is obtained by performing multiple rounds of iterative training on multiple training samples, and each training sample at least includes: an image containing an operation action, and a real label annotated for the image based on the operation action; or, an image containing a device state of a target device, and a real label annotated for the image based on the target device and the device state.

[0022] Optionally, performing target recognition processing on the at least two key frames based on image features in the at least two key frames through a trained target recognition model to obtain corresponding target recognition results includes:

[0023] For each of the at least two key frames, perform the following operations respectively:

[0024] Performing feature extraction processing on the key frame to obtain image features corresponding to the key frame;

[0025] Based on the image channel information and image space information of the key frame in the image features, increasing the attention weight for the target object and the target device in the key frame;

[0026] Based on the improved attention weight and in combination with the image features, the key frame is recognized and processed to obtain a corresponding target recognition result.

[0027] Optionally, the target recognition model is obtained by performing multiple rounds of iterative training on multiple training samples, and the following operations are performed in each round of iterative training:

[0028] Performing target recognition processing on at least one training sample through the target recognition model to be trained to obtain a corresponding sample recognition result;

[0029] Based on the difference between the sample recognition result and the true label corresponding to the at least one training sample, and the loss weight corresponding to the type of the operation action or the device state represented by the true label, obtaining a target loss value corresponding to the current round of iteration process;

[0030] Based on the target loss value, parameters of the target recognition model to be trained are adjusted.

[0031] In a second aspect, an embodiment of the present application provides a device for target recognition, including:

[0032] An acquisition module, used to acquire a video to be identified; the video to be identified is used to record the operation action of the target object on at least one target device, and the device status of the at least one target device;

[0033] A processing module, configured to obtain at least two key frames satisfying a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified; wherein the at least two key frames record: an action change in which the pixel difference corresponding to the target object exceeds a first change threshold and / or a state change in which the pixel difference corresponding to at least one target device exceeds a second change threshold;

[0034] A recognition module, configured to perform target recognition processing on the at least two key frames based on image features in the at least two key frames to obtain corresponding target recognition results; the target recognition results represent: target operation actions corresponding to the target objects in the key frames, and target device states of the at least one target device;

[0035] The evaluation module is used to evaluate the target operation and at least one of the target device states based on the preset corresponding relationship between the target operation action and at least one of the target device states and the evaluation criteria, and obtain the operation evaluation result corresponding to the target object.

[0036] Optionally, the processing module is used to obtain at least two key frames satisfying a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified, specifically to:

[0037] For any two adjacent image frames in the video to be identified, perform the following operations respectively:

[0038] Based on the pixel values ​​corresponding to the two adjacent image frames, a difference operation is performed to obtain the pixel difference corresponding to each pixel position in the two adjacent image frames;

[0039] When the sum of the absolute values ​​of the pixel differences is greater than or equal to the first difference threshold included in the preset pixel condition, a key frame corresponding to the video to be identified is obtained based on the two adjacent image frames.

[0040] Optionally, when the processing module is used to obtain the key frame corresponding to the video to be identified based on the two adjacent image frames, it is specifically used to:

[0041] Based on the numerical relationship between the pixel difference corresponding to each pixel position in the two adjacent image frames and the second difference threshold value included in the preset pixel condition, the pixels in the two adjacent image frames are divided into foreground pixels and background pixels;

[0042] For each of the two adjacent image frames, the following operations are performed respectively: the pixel value corresponding to the background pixel in one image frame is adjusted to a preset pixel value, and the image frame after the pixel value is adjusted is used as a key frame corresponding to the video to be identified.

[0043] Optionally, the recognition module is used to perform target recognition processing on the at least two key frames based on image features in the at least two key frames, and when obtaining corresponding target recognition results, is specifically used to:

[0044] Using a trained target recognition model, based on image features in the at least two key frames, performing target recognition processing on the at least two key frames to obtain corresponding target recognition results;

[0045] The target recognition model is obtained by performing multiple rounds of iterative training on multiple training samples, and each training sample at least includes: an image containing an operation action, and a real label annotated for the image based on the operation action; or, an image containing a device state of a target device, and a real label annotated for the image based on the target device and the device state.

[0046] Optionally, the recognition module is used to perform target recognition processing on the at least two key frames based on the image features in the at least two key frames through a trained target recognition model, and when obtaining corresponding target recognition results, is specifically used to:

[0047] For each of the at least two key frames, perform the following operations respectively:

[0048] Performing feature extraction processing on the key frame to obtain image features corresponding to the key frame;

[0049] Based on the image channel information and image space information of the key frame in the image features, increasing the attention weight for the target object and the target device in the key frame;

[0050] Based on the improved attention weight and in combination with the image features, the key frame is recognized and processed to obtain a corresponding target recognition result.

[0051] Optionally, the device further comprises a training module, wherein the training module is used to perform multiple rounds of iterative training through multiple training samples to obtain the target recognition model, wherein the training module is used in each round of iterative training to specifically:

[0052] Performing target recognition processing on at least one training sample through the target recognition model to be trained to obtain a corresponding sample recognition result;

[0053] Based on the difference between the sample recognition result and the true label corresponding to the at least one training sample, and the loss weight corresponding to the type of the operation action or the device state represented by the true label, obtaining a target loss value corresponding to the current round of iteration process;

[0054] Based on the target loss value, parameters of the target recognition model to be trained are adjusted.

[0055] In a third aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the instructions stored in the memory, so that the at least one processor executes the method described in the first aspect or any optional embodiment of the first aspect.

[0056] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the method in the first aspect or any optional implementation of the first aspect is implemented.

[0057] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions, wherein the computer program product stores instructions that, when executed on a computer, enable the computer to execute the method described in the first aspect or any optional implementation of the first aspect.

[0058] The beneficial effects of this application are as follows:

[0059] In this solution, key frame extraction is performed on the video to be identified that records the operation action of the target object and the device status of the target device, and the change of pixel values ​​between image frames is used to extract the key frames that can record the action change of the target object and the state change of the target device. In this way, a part of the image frames extracted from the video to be identified are used as key frames, which reduces the amount of data that needs to be processed during the target recognition process, thereby improving the processing efficiency of the target recognition process. At the same time, the interference of other unnecessary content on the target recognition process can also be reduced, and the accuracy of the target recognition process can also be improved.

[0060] Furthermore, after the target recognition processing is performed on the key frame, the corresponding target recognition result is obtained to characterize the target operation action of the target object in the key frame, and the target device state corresponding to the target device, so that the preset corresponding relationship between it and the evaluation standard can be used to objectively, quickly and conveniently perform evaluation processing on the target object, and then obtain the corresponding operation evaluation result. In this way, the efficiency of obtaining the operation evaluation result and the objectivity of the operation evaluation result are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 A schematic diagram of possible application scenarios provided for embodiments of the present application;

[0063] Figure 2 A flowchart of a method for target identification provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of a corresponding relationship between pixel positions of adjacent frames provided in an embodiment of the present application;

[0065] Figure 4 A flowchart of a method for obtaining a key frame provided in an embodiment of the present application;

[0066] Figure 5 A logical schematic diagram of a method for obtaining a key frame provided in an embodiment of the present application;

[0067] Fig. 6A A flowchart of a method for obtaining a key frame provided in an embodiment of the present application;

[0068] Figure 6B A schematic diagram of a motion graph when obtaining a key frame provided in an embodiment of the present application;

[0069] Figure 7 A schematic diagram of a training sample provided in an embodiment of the present application;

[0070] Figure 8 A training flow chart of a target recognition model provided in an embodiment of the present application;

[0071] Fig. 9 A flowchart of a method for target recognition processing provided in an embodiment of the present application;

[0072] Fig.10 A schematic diagram of the structure of a device for target recognition is provided for an embodiment of the present application;

[0073] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is only a part of the embodiment of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In the absence of conflict, the embodiments in the present application and the features in the embodiments can be arbitrarily combined with each other. In addition, although the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in an order different from that here.

[0075] The terms "first" and "second" in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive protection. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices. "Multiple" in the present application can mean at least two, for example, two, three or more, and the embodiments of the present application are not limited.

[0076] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0077] In the technical solution of this application, the acquisition, transmission, storage, and use of data are in compliance with the requirements of relevant national laws and regulations.

[0078] With the continuous development of information technology, in assessments involving practical operations, video recording equipment has become more and more popular in assessment venues to record candidates' practical operations and results for subsequent review. However, for these assessments involving practical operations, the evaluation criteria adopt the traditional manual assessment method. For example, in the physical, chemical and biological experiment assessments, when the results are measured, it is necessary to manually observe and evaluate the candidate's experimental operations and experimental results. This results in a high labor cost and time cost to obtain the corresponding assessment results in an assessment, and there will be certain subjective factors that affect the candidate's assessment results. At the same time, the recording of the candidate's operating actions by the video recording equipment can only be used for recording, which will result in a large waste of data.

[0079] In view of this, an embodiment of the present application provides a method for target recognition, which obtains a video to be recognized that records the operating actions of the target object on at least one target device and the device status of at least one target device, and uses the image information of the target object in the video to be recognized to identify and judge the operating actions of the target object, thereby obtaining a corresponding operation evaluation result.

[0080] Specifically, the method proposed in this application has the following steps for the video to be identified: obtaining at least two key frames that meet the preset pixel conditions from the video to be identified, and then performing target identification processing based on these key frames; wherein the content represented by the at least two key frames is: the action change of the target object whose pixel difference exceeds the first change threshold, and / or the state change of at least one target device whose pixel difference exceeds the second change threshold. In this way, by obtaining key frames, the static and different image data existing in the video to be identified can be effectively reduced, thereby reducing the amount of data required for target identification processing, so as to improve the processing efficiency of target identification processing.

[0081] Furthermore, after performing target recognition processing and obtaining the corresponding target recognition results, the operations and states in the target recognition results can be evaluated based on the preset correspondence between the target operation actions corresponding to the target object in the key frames identified in the target recognition results, and the target device state of at least one target device, and the evaluation criteria, so as to obtain the operation evaluation results corresponding to the target object.

[0082] In this way, after the operation action of the target object in the video to be identified is identified and processed, the operation of the target object can be evaluated based on the target recognition result obtained, so that the corresponding operation evaluation result can be obtained without manual evaluation.

[0083] After completing the introduction of the main inventive ideas of the embodiments of the present application, the specific implementation of the target identification method proposed in the embodiments of the present application will be introduced below in combination with some specific embodiments and drawings.

[0084] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0085] See also Figure 1 , which is a schematic diagram of a possible application scenario provided in an embodiment of the present application. In this scenario, a terminal device 101 and a server 102 may be included.

[0086] The terminal device 101 may be a mobile phone, a tablet computer (PAD), a personal computer (PC), a wearable device, a vehicle-mounted terminal, etc. The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0087] The terminal device 101 and the server 102 may be directly or indirectly connected to each other through one or more communication networks 103. The communication network 103 may be a wired network or a wireless network, for example, a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network, or other possible networks, which are not limited in the embodiment of the present invention.

[0088] It should be noted that the target recognition method in the embodiment of the present application can be Figure 1 For example, after the server obtains the video to be identified that records the operation action of the target object through the terminal device, it performs relevant processing of extracting key frames, thereby deleting redundant data in the video to be identified that is useless for the target identification process, and then performs corresponding target identification processing on the acquired key frames to obtain the corresponding target identification result, and obtains the corresponding operation evaluation result based on the target operation action and the target device state in the target identification result.

[0089] It should be noted that Figure 1What is shown is just an example. In fact, the number and communication mode of terminal devices and servers are not limited and are not specifically limited in the embodiments of the present application.

[0090] On the other hand, the target identification method provided by the embodiment of the present application can be applied to a variety of assessments involving practical operations. For example, in the physical, chemical and biological experiment assessment, this solution can first collect the assessment video of the experimental assessment, and propose videos to be identified for different target objects therefrom, and then perform corresponding target identification processing on these videos. For another example, in the vehicle driving assessment, this solution can also collect the operation video of the examinee, and at the same time record the vehicle status corresponding to the vehicle, and combine the target identification processing of the video to be identified for the target object of this solution, and the vehicle status of the operated vehicle, to comprehensively evaluate the examinee. It should be noted that the examples of the above application scenarios are all used to provide possible application scenarios of this solution, rather than specific restrictions.

[0091] The following describes the target identification method provided by the exemplary embodiment of the present application in combination with the application scenario described above and with reference to the accompanying drawings. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this regard.

[0092] See also Figure 2 , is a flow chart of a method for target recognition provided in an embodiment of the present application. In order to facilitate the subsequent introduction of the method, the following will be Figure 1 The method is described by taking the server shown as an execution subject as an example. Figure 2 As shown, the specific implementation steps of this method are as follows:

[0093] Step S201: Acquire a video to be identified; wherein the video to be identified is used to record the operation actions of the target object on at least one target device, and the device status of at least one target device.

[0094] Before introducing this scheme in detail, some terms proposed in this method are first introduced.

[0095] First, the acquired video to be identified is a video recorded with the target object after recording the target object, and records the operation process of the target object. In the video, the target object's operation action on at least one target device and the device status of at least one target device are recorded.

[0096] Secondly, the target object can be any candidate in the different practical assessments as mentioned above. For example, the target object can be a candidate in a physical, chemical and biological experiment assessment, a candidate in a vehicle driving test, a candidate in a physician qualification examination, etc., and this application does not limit this. When the target object is conducting these possible assessments, it will operate at least one target device, such as a voltmeter, weights, steering wheel, tweezers, etc. Therefore, the device status of at least one of these target devices will also be recorded in the video to be identified.

[0097] As for the video to be identified, not every frame of the image contained therein is necessarily an image that records valid information. For example, during the operation of a certain candidate, there is a certain thinking process. In this case, during the static thinking process, the recorded video content will not have a high recognition value.

[0098] Therefore, for the server, after obtaining the video to be identified, it can continue to perform the following operations:

[0099] Step S202: based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified, at least two key frames satisfying a preset pixel condition are obtained from the video to be identified.

[0100] Among them, at least two key frames record: an action change in which a pixel difference corresponding to the target object exceeds a first change threshold, and / or a state change in which a pixel difference corresponding to at least one target device exceeds a second change threshold.

[0101] When the server obtains the key frame in the video to be identified, it can use the pixel difference between the corresponding pixel positions between adjacent frames to determine. Figure 3 As shown, assuming that there are adjacent i-th frame image and i+1-th frame image, and a square in the image represents a pixel in the image. Then, the pixel difference proposed in step S202 refers to the difference between the pixel values ​​corresponding to the same position in the i-th frame image and the i+1-th frame image.

[0102] By utilizing the pixel difference between adjacent frames, the server can combine preset pixel conditions to extract at least two corresponding key frames from the video to be identified, thereby obtaining key frame data that can record the motion changes of the target object and / or the state changes of at least one target device.

[0103] The basis for determining whether the target object has undergone a change in motion or a change in the state of the target device may be that the pixel difference between adjacent frames corresponding to the pixel of the target object in the video to be identified exceeds a first change threshold. In this way, when the pixel value corresponding to the pixel changes by more than a certain range, the server can determine that the target object corresponding to the pixel may have undergone a certain change in motion.

[0104] Alternatively, it may be the pixel corresponding to the target device in the video to be identified. When the pixel difference between adjacent frames exceeds the second change threshold, and the change amplitude of the pixel corresponding to the target device also exceeds a certain range, the server can also determine that the target device corresponding to the pixel may have a state change.

[0105] Optionally, for the first change threshold and the second change threshold mentioned above, they are threshold contents pre-set in the preset pixel conditions, and for their specific values, since the positive and negative values ​​of the pixel difference cannot be unified, therefore, for the first change threshold and the second change threshold, their specific values ​​can be a data range rather than a number.

[0106] As for the method of obtaining at least two corresponding key frames from the video to be identified, the server can complete it in the following way:

[0107] In a possible implementation, the server may perform the following operations respectively for any two adjacent image frames in the video to be identified:

[0108] See also Figure 4 , is a flowchart of a method for obtaining a key frame provided in an embodiment of the present application, such as Figure 4 As shown, the specific implementation steps of this method are as follows:

[0109] Step S401: performing a difference operation based on the pixel values ​​corresponding to each of two adjacent image frames to obtain the pixel difference corresponding to each pixel position in the two adjacent image frames.

[0110] Specifically, there will be a relatively obvious difference between two adjacent image frames. Therefore, by subtracting the two image frames, the pixel difference corresponding to each pixel position in the two adjacent image frames can be obtained.

[0111] Step S402: when the sum of the absolute values ​​of the pixel differences is greater than or equal to the first difference threshold included in the preset pixel condition, a key frame corresponding to the video to be identified is obtained based on the two adjacent image frames.

[0112] After the server obtains the pixel difference corresponding to each pixel position, it can further obtain the sum of the absolute values ​​corresponding to the pixel difference, and judge the size of the difference between the two adjacent image frames from the sum of the absolute values. Exemplarily, when the sum of the corresponding absolute values ​​is greater than or equal to the first difference threshold included in the preset pixel condition, it indicates that the image represented by the two adjacent pixel frames has a large change, and it also indicates that the target object may have a change in action in the two image frames, or the target device has a change in device status. When the sum of the corresponding absolute values ​​is less than the first difference threshold, it can be represented that the target object and target device corresponding to the two image frames have no significant changes.

[0113] In this way, after clarifying the numerical relationship between the sum of the absolute values ​​of the pixel differences between two adjacent image frames and the first difference threshold, the server can obtain the key frame corresponding to the video to be identified based on the two adjacent pixel frames whose absolute value sum is greater than or equal to the first difference threshold.

[0114] Specifically, the server may obtain the key frame in the following different ways.

[0115] Method 1: Obtain directly.

[0116] For the server, when it is determined that the sum of the absolute values ​​of the pixel differences between two adjacent image frames is greater than or equal to the first difference threshold, the server can directly use the two adjacent image frames as two frames in the key frames corresponding to the video to be identified.

[0117] In this way, when generalizing to all two adjacent frames in the video to be identified, Figure 5 As shown, the server can directly select image frames that may record changes in target objects and target devices from all video frames as key frames, and delete those image frames that record no significant changes as redundant data of the video to be identified, thereby reducing the amount of data required for target identification processing, thereby improving the efficiency of target identification processing.

[0118] As for the video to be identified, since the framing range does not necessarily contain only the target object when it is recorded, it may also contain scene information during the practical test and other content. Therefore, for the target recognition processing, some background information will also interfere with the efficiency of the target recognition processing.

[0119] Therefore, in view of this situation, when obtaining the key frame, the server can also perform the following methods:

[0120] Method 2: Partial acquisition.

[0121] When the server has determined that the sum of the absolute values ​​of the pixel differences between the two adjacent image frames is greater than or equal to the first difference threshold, the server may further perform certain processing on the two image frames and then obtain corresponding key frames.

[0122] For details, see Fig. 6A , is a flowchart of a method for obtaining a key frame provided in an embodiment of the present application, such as Fig. 6A As shown, the specific implementation steps of this method are as follows:

[0123] Step S601: Based on the numerical relationship between the pixel difference corresponding to each pixel position in the two adjacent image frames and the second difference threshold included in the preset condition, the pixels in the two adjacent image frames are divided into foreground pixels and background pixels.

[0124] For the two adjacent pixel frames, the server may determine the numerical relationship between the absolute value of the pixel difference at each pixel position and the second difference threshold value.

[0125] When the absolute value of the pixel difference is greater than or equal to the second difference threshold, the server can classify the pixel position corresponding to the pixel difference in the two image frames as a foreground pixel, and when the absolute value of the pixel difference is less than the second difference threshold, the server can classify the pixel position corresponding to the pixel difference in the two image frames as a background pixel.

[0126] In this way, after the server classifies the type of each pixel in the image frame based on the numerical relationship between the absolute value of the pixel difference and the second difference threshold, the server can perform the following operations for each image frame:

[0127] Step S602: adjusting the pixel values ​​corresponding to the background pixels in an image frame to preset pixel values, and using the image frame after the pixel values ​​are adjusted as a key frame corresponding to the video to be identified.

[0128] After the server divides the foreground pixels and the background pixels, it can adjust the pixels in the pixel frame accordingly based on this division method. The reason is that since the background pixels are pixels with a smaller degree of pixel value change, the information recorded by these pixels is more likely to be irrelevant to the target object and the target device. For example, these background pixels may record the contents of a table, a wall, etc.

[0129] Therefore, in order to improve the efficiency of subsequent target recognition, the server can adjust the pixel values ​​corresponding to the background pixels in the image frame to preset pixel values, and then use the image frame with adjusted pixel values ​​as one of the key frames corresponding to the video to be recognized.

[0130] The preset pixel value may be set to zero or other fixed values, so that the server can directly ignore these background pixels in the subsequent recognition process and perform corresponding processing on the foreground pixels, thereby improving the efficiency of the target recognition process.

[0131] Optionally, for the step of adjusting the pixel values ​​corresponding to the background pixels, another implementation method may be to set a corresponding mask matrix for the divided background pixels, and use the matrix elements in the mask matrix to correspond to the positions of the background pixels in the image frame, so that in the subsequent target recognition processing, only the foreground pixels can be processed based on the mask matrix.

[0132] In this method, two adjacent image frames that meet the first difference threshold are further subjected to pixel adjustment processing. While reducing the image frames that require target recognition processing, the pixel values ​​to be identified contained in the key frames are further reduced. In this way, the amount of data required for target recognition processing can be reduced, thereby improving the efficiency of obtaining target recognition results.

[0133] In one possible implementation, when acquiring key frames, the window for calculating the difference between corresponding pixels can be dynamically adjusted according to the video length of the video to be identified, and the window size can be set to be positively correlated with the video length, so as to avoid the situation where the length of the video to be identified is short but the window is too large, resulting in the extracted key frames ignoring the key operations of the target object or the key changes of the target device.

[0134] For example, suppose there is a video to be identified, which has a total of 2696 frames, which corresponds to a duration of about two and a half minutes. At this time, if the window value is set to 30, the following can be obtained: Figure 6B The schematic diagram of the motion graph shown. Among them, the horizontal axis represents the current frame, and the vertical axis represents the sum of the absolute values ​​of the pixel differences between two adjacent frames within the window value. In this way, by setting a suitable first difference threshold, it is possible to determine whether these image frames are key frames. Among them, the setting of the first difference threshold can be set based on experience according to the content, scene, lighting and other conditions in the video to be identified, or it can be set by a pre-trained model, and the present application does not impose any restrictions on this. The above introduces several methods for obtaining at least two key frames, and after the server obtains at least two key frames corresponding to the video to be identified, it can continue to perform the following steps for these key frames:

[0135] Step S203: Based on the image features in the at least two key frames, perform target recognition processing on the at least two key frames to obtain corresponding target recognition results.

[0136] The target recognition result represents: a target operation action corresponding to a target object in a key frame, and a target device state corresponding to at least one target device.

[0137] For the server, after identifying the key frames, the target recognition results obtained are characterized by some target operation actions of the target object in the video to be identified, as well as the target device status. These target operation actions can be pre-set in the server, actions that must appear in the practical assessment, or actions that must not appear. Similarly, the target device status can also be pre-set in the server, actions that must appear or must not appear. As for other operation actions and device status, they can be determined as non-target operation actions or non-target device status.

[0138] In one possible implementation, when the server performs target recognition processing on these key frames, it can use a trained target recognition model pre-set in the server, that is: after the server inputs the key frame data into the target recognition model, the target recognition model is used to perform target recognition processing on the at least two key frames based on image features in the at least two key frames, thereby obtaining corresponding target recognition results.

[0139] The target recognition model may be obtained through multiple training samples and multiple rounds of iterative training, and each training sample at least includes: an image containing an operation action, and a real label annotated for the image based on the operation action. Alternatively, the training sample may also include: an image containing a device state of a target device, and a real label annotated for the image based on the target device and the device state.

[0140] For example, Figure 7 As shown, for a training sample, it may include the operation of the target object picking up a measuring cup. For this operation, the real label corresponding to the training sample can be: target operation object 1, or a label such as picking up a measuring cup, to represent the real content contained in the image.

[0141] As for the acquisition of these training samples, dedicated hardware equipment can be used to unify the viewing angle range and lens height, and collect videos in simulated practical assessments. Among them, the unified viewing angle can be a video acquisition with the target object as the first viewing angle, or it can be a viewing angle facing the target object from above. The specific viewing angle is determined according to the use requirements, and this application does not limit this. For example, a simulated experimental test can be performed to collect videos to ensure that the data fits the actual scene. In this way, the real labels corresponding to the training samples collected by measuring resistance by volt-ampere method as an example can include: target devices such as ammeter, voltmeter, light bulb, sliding rheostat, power supply, and whether the terminal is connected to the wire, whether the slider, whether the switch is turned on, whether there is zeroing, etc. The device status changes on the device. In this way, after selecting certain image frames from the collected video and annotating them with the above-mentioned labels, multiple training samples required for training the target recognition model can be obtained.

[0142] In this way, by performing multiple rounds of iterative training on the target recognition model through a variety of training samples, the target recognition model can learn the feature content corresponding to different target operation actions, target devices and target device states, and then quickly perform target recognition processing on them.

[0143] As for the training process of the target recognition model, it can be carried out in the following way:

[0144] In a possible implementation, for the target recognition model to be trained, the server needs to perform multiple rounds of iterative training on the target recognition model using the above-mentioned multiple training samples, wherein the server needs to perform the following operations:

[0145] See also Figure 8 , is a training flow chart of the target recognition model provided in the embodiment of the present application, which includes the following steps:

[0146] Step S801: Acquire multiple training samples.

[0147] Among them, the multiple training samples mentioned above are the multiple training samples proposed above, and this application will not elaborate on them here.

[0148] Step S802: performing target recognition processing on at least one training sample of this iteration through the target recognition model to be trained in this iteration to obtain a corresponding sample recognition result.

[0149] In the embodiment of the present application, in each round of iterative training, all training samples can be used as the input of this round, or part of the training samples can be extracted as the input of this round. The target recognition process will be described in detail later, so it will not be described in detail here.

[0150] Step S803: Based on the sample recognition result of each training sample and the corresponding true label, determine the target loss value of the target recognition model to be trained in this iteration.

[0151] Among them, the target loss value can be calculated using any possible loss function, such as the cross entropy loss function, whose formula is: Among them, y represents the true label, y ^ Characterize the prediction probability, or mean squared error (MSE) loss function, square absolute error loss function, maximum likelihood (LHL) loss function, etc., and can also use focal loss function (FocalLoss), target detection task (Complete Intersection over Union, CIoU) loss function, etc. Of course, it can also be other possible loss functions, and the embodiment of the present application does not limit this.

[0152] In a possible implementation, when the loss value is obtained during any round of training, the server may further perform the following operations, namely:

[0153] Based on the difference between the sample recognition result and the true label corresponding to at least one training sample, and the loss weight corresponding to the type of operation action or device state represented by the true label, the target loss value corresponding to this round of iteration process is obtained.

[0154] When obtaining the sample recognition result, it is necessary to obtain the real label corresponding to the training sample at the same time to obtain the difference between the two. However, for some types of target operation actions or target device states, the number of corresponding training samples obtained is not comparable to the number of training samples of other types, which leads to the problem that the training samples for different types of operations are unbalanced.

[0155] To this end, corresponding loss weights are set in advance for different types to adjust the impact of different types of training samples on the final loss value.

[0156] For example, it is usually much easier to obtain training samples for non-target operation actions than for target operation actions. Therefore, the number of samples corresponding to non-target operation actions will greatly exceed the samples corresponding to target operation actions, which leads to an imbalance between positive and negative samples. Therefore, corresponding weights can be set in advance for the types of positive and negative samples, such as a weight of 0.25 for negative samples, etc., to reduce the impact of negative samples on the loss value, thereby improving the recognition accuracy of the trained target recognition model for different types of objects.

[0157] Step S804: Determine whether the model to be trained in this iteration has reached the convergence condition.

[0158] In the embodiment of the present application, the convergence condition may include one or more of the following conditions:

[0159] (1) The number of iterations reaches the set threshold.

[0160] (2) The target loss value is less than the set loss threshold.

[0161] Step S805: If the determination process of step S804 is no, then based on the target loss value, the parameters of the target recognition model to be trained are adjusted.

[0162] In the embodiment of the present application, when the number of iterations does not exceed the preset number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 804 can be determined to be no, that is, it is considered that the current model to be trained does not meet the convergence condition, and the model parameters need to be adjusted to continue training. After the parameter adjustment, the next iterative training process is entered, that is, jumping to step 802.

[0163] In a possible implementation, when the model still does not meet the convergence condition, the model weight parameters can be updated by the gradient descent method, and the training can be continued with the updated model weight parameters.

[0164] When the number of iterations has exceeded the preset threshold, or the model loss value is less than the set loss threshold, the determination process of step 804 is yes, that is, it is considered that the current recognition model to be trained has met the convergence conditions, the model training is completed, and the keyword recognition model is obtained.

[0165] In order to verify the effect of the trained keyword recognition model, the embodiment of the present application tests the target recognition model using the verification data in the verification set. The verification set can be at least one training sample that does not participate in the training process among the above-mentioned multiple training samples, or it can be other sample data with real labels in addition to the training samples, and the present application does not limit this.

[0166] If the evaluation index reaches the preset evaluation effect, the target recognition model to be trained can be used in the actual target recognition process. If the model evaluation fails, training and testing need to be continued until the target recognition model to be trained reaches the preset evaluation effect.

[0167] In a possible implementation, when the trained target recognition model is applied to an actual target recognition method, when performing target recognition processing on at least two key frame images, the following operation steps may be performed for each of the at least two key frames:

[0168] See also Fig. 9 , is a flow chart of a method for target recognition processing provided by an embodiment of the present application, such as Fig. 9 As shown, the specific implementation steps of this method are as follows:

[0169] Step S901: performing feature extraction processing on the key frame to obtain image features corresponding to the key frame.

[0170] When the server uses the target recognition model to extract image features, it can use a convolutional neural network to extract high-dimensional features corresponding to the image as image features.

[0171] Step S902: Based on the image channel information and image space information of the key frame in the image features, the attention weight of the target object and the target device in the key frame is increased.

[0172] For the server, after extracting the image features, the Convolutional Block Attention Module (CBAM) can be used to process the image features. The channel attention and spatial attention mechanisms are used to extract the image channel information and image spatial information contained in the image features, and the channel and spatial weight information in the image features are adjusted to increase the attention weight for the target objects and target devices in the key frames. In this way, the feature expression corresponding to the target objects and target devices is improved, and the perception ability of the target recognition model for the target objects and target devices is also enhanced.

[0173] Step S903: Based on the increased attention weight and in combination with the image features, the key frames are recognized and processed to obtain corresponding target recognition results.

[0174] After increasing the attention weights for the target object and target device, the target recognition model can pay more attention to the image features of the target object and target device, thereby improving the efficiency and effectiveness of the recognition processing for these image features.

[0175] The above introduces some possible implementation methods that can be adopted when training the target recognition model. In order to facilitate understanding of the process of model training and application, the following will introduce this method with a possible example.

[0176] Exemplarily, for the target recognition model, the eighth-generation target detection algorithm (You Only Look Once, YOLO) (abbreviated as: yolov8s) can be selected as the detection network basis, and further improvements can be made by adding CBAM to the neck network of the model to enhance the network results, and then the multiple training samples proposed above are used for training to obtain the final target recognition model.

[0177] Specifically, in the backbone network stage of the model, it mainly includes the convolution (Conv) module, the bottleneck (CSPBottleneck with 2convolutions, C2F) module and the fast spatial pyramid pooling (Spatial PyramidPooling–Fast, SPPF) module.

[0178] Among them, the Conv module includes convolution (Conv2d), batch normalization (BN) and activation function (SiLU).

[0179] The C2F module uses three convolution blocks with kernel_size=1, stride=1 and n Bottleneck modules, and adjusts the number of channels for different scale models. The Bottleneck module first performs two convolution layers with kernel_size=3, stride=1, padding=1, and then adds them to the input feature map to achieve feature fusion.

[0180] The SPPF module uses multiple small-size pooling kernels in cascade to perform 5×5, 9×9, and 13×13 maximum pooling on the image to extract features from multiple aspects, and then concatenates the three groups of pooled features with the initial image.

[0181] Select yolov8s as the basic detection network for improvement, and set the scaling factor of the Conv channel d = 0.33, and the layer scaling factor of the BottleneckCSP module w = 0.50. Input a picture of size 640×640×3, first replace the original focus module with a convolution kernel with kernel_size of 6, stride of 2, and padding of 2, and get a feature map of 320×320×2.

[0182] In one possible implementation, the feature extraction steps proposed above are as follows:

[0183] First, input a picture of size 640×640×3, and first pass it through a convolutional layer with kernel_size=3, stride=2, and padding=2 to obtain a feature map of 320×320×64;

[0184] Secondly, after two consecutive passes through the Conv and C3 combination modules, the first 80×80×256×w large-scale feature map (Layer4) output is obtained. The number of Bottleneck modules n in C2F is set to 3×d;

[0185] Then, the second 40×40×512×w medium-scale feature map (Layer6) is obtained through another Conv and C3 combination module. The number of Bottleneck modules n in C2F is set to 6×d;

[0186] Finally, through a Conv and C3 combination module and SPPF network layer, the third 20×20×1024×w small-scale feature map (Layer9) is obtained. The number of Bottleneck modules n in C2F is set to 3×d.

[0187] In the Neck layer of the model, Feature Pyramid Networks (FPN) and Pyramid Attention Networks (PAN) structures are used. FPN is transmitted and fused through top-down upsampling, and the PAN structure is responsible for the bottom-up transmission of image detail information. Each stage is downsampled by 2 times through 3×3 convolution, and then added and fused with the feature map of the corresponding size in FPN. The Bottleneck module in the C2F module is different from that in Backbone. It directly performs two 3×3 convolution outputs instead of a residual network. The CBAM module uses channel attention and spatial attention mechanisms to dynamically adjust the channel weights and spatial importance of the feature map to enhance the perception of the target.

[0188] Assume that the input to CBAM is defined as Where C, H, and W represent the channels, height, and width of the input features.

[0189] F in They are processed by global average pooling AvgPool and global maximum pooling MaxPool respectively. Therefore, two sets of processed feature maps are obtained. Then, these feature maps are passed through a multi-layer perceptron MLP with only one hidden layer. We get a size of These features are added and activated using the sigmoid function, and the output F of the channel attention (CAM) module is x Defined as:

[0190]

[0191] Next, the spatial attention (SAM) module takes the output of the CAM module as input, and also undergoes global maximum pooling and global average pooling to obtain two H×W×1 images spliced ​​in the channel direction. After passing through a 7×7 convolutional layer and a sigmoid function, we get the final CBAM output feature.

[0192]

[0193] Finally, the feature output is multiplied by the original input feature of the module as the final output, as follows:

[0194]

[0195] In one possible implementation, the above steps are as follows:

[0196] First, Layer9 is upsampled and concat with Layer6, and then passes through the C2F module and CBAM module to obtain the output 40×40×512×w feature map (Layer12);

[0197] Secondly, Layer12 is further upsampled, concatted with Layer4, and then passed through the C2F module to obtain the first 80×80×256×w output feature map of the Neck layer (Layer15);

[0198] Next, Layer15 is convolved with 3×3 to perform 2x downsampling, and then concat with the Layer12 feature map. After passing through the C2F module and the CBAM module, the second 40×40×512×w output feature map (Layer18) of the Neck layer is obtained.

[0199] Finally, Layer18 is downsampled by 2 times through 3×3 convolution, and then concat with the Layer9 feature map. After passing through the C2F module and the CBAM module, the third 20×20×1024×w output feature map (Layer21) of the Neck layer is obtained.

[0200] In the head layer of the model, for each feature map of different scales obtained in the neck (Layer15, Layer18, Layer21), the decoupled head structure is used to separate the classification and detection heads, and the positive and negative sample allocation and loss calculation are performed. Using the anchor-free frame, one head is used for target recognition and the other head is used for classification.

[0201] First, the positive and negative samples are assigned using a dynamic assignment strategy (Task-Aligned Assigner), which dynamically adjusts the distribution ratio of positive and negative samples during training. The formula is as follows:

[0202] t=s α ×u β (Formula 4)

[0203] Where s and y are the classification score and IOU value respectively, and α and β are weight hyperparameters. The multiplication of the two can measure the degree of alignment. t can simultaneously control the optimization of the classification score and IOU to achieve task alignment, which can guide the network to dynamically focus on high-quality aiming frame anchors.

[0204] Secondly, the loss calculation includes two branches: classification and regression. The classification branch uses the focal loss function to solve the problem of extremely unbalanced sample numbers. The regression branch uses CIoU Loss. The total loss formula is as follows:

[0205]

[0206] Among them, N is the number of detection layers, B is the number of targets whose labels are assigned to the prior box L CIoU , L fl CIoULoss and Focal Loss respectively, given a prediction box b and a target box b gt , CIoU loss is defined as:

[0207]

[0208] in,

[0209]

[0210] b、b gt They are the prediction box and the label box respectively, and v is used to measure the consistency of the relative proportions of the two rectangular boxes. is the weight coefficient, ρ represents the Euclidean distance between two rectangular boxes, and w gt 、h gt , w, and h are the width and height of the label box and the prediction box respectively.

[0211] The focal loss function is defined as:

[0212]

[0213] The parameter α is added, generally set to 0.25, which simply reduces the weight of positive or negative samples to solve the problem of sample imbalance.

[0214] Finally, after getting the prediction of the target, NMS is used to suppress non-maximum targets and remove redundancy. The main thing is to sort the confidence of each category from large to small and select the bounding box with the highest confidence.

[0215] The above describes the training process of the target recognition model and the target recognition processing process of the target recognition model for at least two key frames. After the target recognition model outputs the corresponding target recognition result, the server can continue to perform the following operations:

[0216] Step S204: Based on the preset correspondence between the target operation action and at least one target device state and the evaluation criteria, the target operation action and at least one target device state are evaluated and processed to obtain an operation evaluation result corresponding to the target object.

[0217] After obtaining the target recognition result, the server can evaluate the target operation action and at least one target device state based on the preset corresponding relationship. For example, assuming that there are operation actions 1, 2, and 3 in the target recognition result, as well as device states 1, 2, and 3, and the server has preset the evaluation criteria corresponding to these actions and states, then the server can evaluate them according to these evaluation criteria. For example, if operation action 1 appears, 3 points are added to the target object, and if operation action 2 appears, 4 points are added to the target object. However, if operation action 3 appears, 10 points need to be reduced for the target object, and so on. The same is true for the device state. In the target recognition result, if device states 1, 2, and 3 appear, 2 points can be added to the target object respectively. In this way, the target object is given the final operation evaluation result according to the target operation action and at least one target device state of the target object identified in the target recognition result.

[0218] In this solution, key frame extraction is performed on the video to be identified that records the operation action of the target object and the device status of the target device, and the change of pixel values ​​between image frames is used to extract the key frames that can record the action change of the target object and the state change of the target device. In this way, a part of the image frames extracted from the video to be identified are used as key frames, which reduces the amount of data that needs to be processed during the target recognition process, thereby improving the processing efficiency of the target recognition process. At the same time, the interference of other unnecessary content on the target recognition process can also be reduced, and the accuracy of the target recognition process can also be improved.

[0219] Furthermore, after the target recognition processing is performed on the key frame, the corresponding target recognition result is obtained to characterize the target operation action of the target object in the key frame and the target device state corresponding to the target device, so that the preset correspondence between it and the evaluation standard can be used to objectively, quickly and conveniently perform evaluation processing on the target object, and then obtain the corresponding operation evaluation result. In this way, the efficiency of obtaining the operation evaluation result and the objectivity of the operation evaluation result are improved.

[0220] Based on the same inventive concept, an embodiment of the present application also provides a device for target identification.

[0221] See also Fig.10 , which is a schematic diagram of the structure of a target identification device provided in an embodiment of the present application. The device may be the above-mentioned server or a chip or integrated circuit in the device, etc. The device includes a module / unit / technical means for executing the method executed by the server in the above-mentioned method embodiment.

[0222] Exemplarily, the device 1000 includes:

[0223] The acquisition module 1001 is used to acquire a video to be identified; the video to be identified is used to record the operation action of the target object on at least one target device, and the device status of at least one target device;

[0224] The processing module 1002 is used to obtain at least two key frames that meet a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified; wherein the at least two key frames record: an action change in which the pixel difference corresponding to the target object exceeds a first change threshold and / or a state change in which the pixel difference corresponding to at least one target device exceeds a second change threshold;

[0225] The recognition module 1003 is used to perform target recognition processing on at least two key frames based on image features in at least two key frames to obtain corresponding target recognition results; the target recognition results represent: target operation actions corresponding to the target objects in the key frames, and target device states of at least one target device;

[0226] The evaluation module 1004 is used to evaluate the target operation and at least one target device state based on the preset corresponding relationship between the target operation action and at least one target device state and the evaluation criteria, and obtain the operation evaluation result corresponding to the target object.

[0227] Optionally, the processing module 1002 is used to obtain at least two key frames satisfying a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified, specifically to:

[0228] For any two adjacent image frames in the video to be identified, perform the following operations respectively:

[0229] Based on the pixel values ​​corresponding to the two adjacent image frames, a difference operation is performed to obtain the pixel difference corresponding to each pixel position in the two adjacent image frames;

[0230] When the sum of the absolute values ​​of the pixel differences is greater than or equal to the first difference threshold included in the preset pixel condition, a key frame corresponding to the video to be identified is obtained based on two adjacent image frames.

[0231] Optionally, when the processing module 1002 is used to obtain a key frame corresponding to the video to be identified based on two adjacent image frames, it is specifically used to:

[0232] Based on the numerical relationship between the pixel difference corresponding to each pixel position in the two adjacent image frames and the second difference threshold value included in the preset pixel condition, the pixels in the two adjacent image frames are divided into foreground pixels and background pixels;

[0233] For each of two adjacent image frames, the following operations are performed respectively: the pixel value corresponding to the background pixel in one image frame is adjusted to a preset pixel value, and the image frame after the pixel value is adjusted is used as a key frame corresponding to the video to be identified.

[0234] Optionally, the recognition module 1003 is used to perform target recognition processing on at least two key frames based on image features in at least two key frames, and when obtaining corresponding target recognition results, specifically to:

[0235] Using the trained target recognition model, based on the image features in the at least two key frames, performing target recognition processing on the at least two key frames to obtain corresponding target recognition results;

[0236] Among them, the target recognition model is obtained by performing multiple rounds of iterative training on multiple training samples, and each training sample at least includes: an image containing an operation action, and a real label annotated for the image based on the operation action; or, an image containing a device state of a target device, and a real label annotated for the image based on a target device and a device state.

[0237] Optionally, the recognition module 1003 is used to perform target recognition processing on at least two key frames based on image features in at least two key frames through a trained target recognition model, and to obtain corresponding target recognition results, specifically for:

[0238] For each of the at least two keyframes, perform the following operations:

[0239] Perform feature extraction processing on the key frames to obtain image features corresponding to the key frames;

[0240] Based on the image channel information and image space information of the key frame in the image features, the attention weight for the target object and the target device in the key frame is improved;

[0241] Based on the improved attention weight and combined with image features, the key frames are recognized and processed to obtain the corresponding target recognition results.

[0242] Optionally, the device further includes a training module 1005, which is used to perform multiple rounds of iterative training using multiple training samples to obtain a target recognition model, wherein the training module 1005 is used in each round of iterative training to specifically:

[0243] Performing target recognition processing on at least one training sample through the target recognition model to be trained to obtain a corresponding sample recognition result;

[0244] Based on the difference between the sample recognition result and the true label corresponding to at least one training sample, and the loss weight corresponding to the type of operation action or device state represented by the true label, a target loss value corresponding to the current round of iteration process is obtained;

[0245] Based on the target loss value, the parameters of the target recognition model to be trained are adjusted.

[0246] As an example, Fig.10 The apparatus described can be used to perform Figure 2 Therefore, for the functions that can be realized by each functional module of the device, reference can be made to Figure 2 The description of the illustrated embodiment will not be repeated here.

[0247] It should be noted that, although several modules or submodules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0248] As a possible product form of the above device, see Fig.11 The embodiment of the present application further provides an electronic device 1100, including:

[0249] At least one processor 1101; and a communication interface 1103 communicatively connected to the at least one processor 1101; the at least one processor 1101 executes instructions stored in the memory 1102, so that the electronic device 1100 executes the method steps performed by any device in the above method embodiments through the communication interface 1103.

[0250] Optionally, the memory 1102 is located outside the electronic device 1100 .

[0251] Optionally, the electronic device 1100 includes the memory 1102, the memory 1102 is connected to the at least one processor 1101, and the memory 1102 stores instructions that can be executed by the at least one processor 1101. Fig.11 The dashed lines indicate that the memory 1102 is optional for the electronic device 1100 .

[0252] The processor 1101 and the memory 1102 may be coupled via an interface circuit or may be integrated together, which is not limited here.

[0253] The specific connection medium between the processor 1101, the memory 1102 and the communication interface 1103 is not limited in the embodiment of the present application. Fig.11 In the embodiment, the processor 1101, the memory 1102 and the communication interface 1103 are connected via a bus 1104. Fig.11 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 In the example, only one thick line is used to represent the bus, but it does not mean that there is only one bus or one type of bus. It should be understood that the processor mentioned in the embodiment of the present application can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which is implemented by reading the software code stored in the memory.

[0254] Exemplarily, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0255] It should be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DirectRambus RAM, DR RAM).

[0256] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) can be integrated into the processor.

[0257] It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0258] As another possible product form, an embodiment of the present application also provides a computer-readable storage medium, which is used to store instructions. When the instructions are executed, the computer executes the method steps executed by any device in the above method example.

[0259] As another possible product form, an embodiment of the present application also provides a computer program product including instructions, wherein the computer program product stores instructions that, when executed on a computer, enable the computer to execute the method steps executed by any device in the above method embodiment.

[0260] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0261] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0262] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0263] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0264] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for target recognition, characterized in that: include: Get the video to be identified; The video to be identified is used to record the operation action of the target object on at least one target device, and the device status of the at least one target device; Based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified, at least two key frames satisfying a preset pixel condition are obtained from the video to be identified; wherein the at least two key frames record: an action change in which the pixel difference corresponding to the target object exceeds a first change threshold and / or a state change in which the pixel difference corresponding to at least one target device exceeds a second change threshold; Based on the image features in the at least two key frames, target recognition processing is performed on the at least two key frames to obtain corresponding target recognition results; the target recognition results represent: the target operation action corresponding to the target object in the key frames, and the target device state of the at least one target device; Based on the preset corresponding relationship between the target operation action and at least one of the target device states and the evaluation criteria, the target operation and at least one of the target device states are evaluated and processed to obtain the operation evaluation result corresponding to the target object.

2. The method according to claim 1, characterized in that The step of obtaining at least two key frames satisfying a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified includes: For any two adjacent image frames in the video to be identified, perform the following operations respectively: Based on the pixel values ​​corresponding to the two adjacent image frames, a difference operation is performed to obtain a pixel difference corresponding to each pixel position in the two adjacent image frames; When the sum of the absolute values ​​of the pixel differences is greater than or equal to the first difference threshold included in the preset pixel condition, a key frame corresponding to the video to be identified is obtained based on the two adjacent image frames.

3. The method according to claim 2, characterized in that The step of obtaining a key frame corresponding to the video to be identified based on the two adjacent image frames includes: Based on the numerical relationship between the pixel difference corresponding to each pixel position in the two adjacent image frames and the second difference threshold value included in the preset pixel condition, the pixels in the two adjacent image frames are divided into foreground pixels and background pixels; For each of the two adjacent image frames, the following operations are performed respectively: the pixel value corresponding to the background pixel in one image frame is adjusted to a preset pixel value, and the image frame after the pixel value is adjusted is used as a key frame corresponding to the video to be identified.

4. The method according to claim 1, characterized in that The performing target recognition processing on the at least two key frames based on the image features in the at least two key frames to obtain corresponding target recognition results includes: Using a trained target recognition model, based on image features in the at least two key frames, performing target recognition processing on the at least two key frames to obtain corresponding target recognition results; The target recognition model is obtained by performing multiple rounds of iterative training on multiple training samples, and each training sample at least includes: an image containing an operation action, and a real label annotated for the image based on the operation action; or, an image containing a device state of a target device, and a real label annotated for the image based on the target device and the device state.

5. The method according to claim 4, characterized in that The method of performing target recognition processing on the at least two key frames based on the image features in the at least two key frames through the trained target recognition model to obtain corresponding target recognition results includes: For each of the at least two key frames, perform the following operations respectively: Performing feature extraction processing on the key frame to obtain image features corresponding to the key frame; Based on the image channel information and image space information of the key frame in the image features, increasing the attention weight for the target object and the target device in the key frame; Based on the improved attention weight and in combination with the image features, the key frame is recognized and processed to obtain a corresponding target recognition result.

6. The method according to claim 4, characterized in that The target recognition model is obtained by performing multiple rounds of iterative training on multiple training samples. In each round of iterative training, the following operations are performed: Performing target recognition processing on at least one training sample through the target recognition model to be trained to obtain a corresponding sample recognition result; Based on the difference between the sample recognition result and the true label corresponding to the at least one training sample, and the loss weight corresponding to the type of the operation action or the device state represented by the true label, obtaining a target loss value corresponding to the current round of iteration process; Based on the target loss value, parameters of the target recognition model to be trained are adjusted.

7. A device for target recognition, characterized in that: include: An acquisition module is used to acquire the video to be identified; The video to be identified is used to record the operation action of the target object on at least one target device, and the device status of the at least one target device; A processing module, configured to obtain at least two key frames satisfying a preset pixel condition from the video to be identified based on the pixel difference corresponding to each pixel position between adjacent frames in the video to be identified; wherein the at least two key frames record: an action change in which the pixel difference corresponding to the target object exceeds a first change threshold and / or a state change in which the pixel difference corresponding to at least one target device exceeds a second change threshold; A recognition module, configured to perform target recognition processing on the at least two key frames based on image features in the at least two key frames to obtain corresponding target recognition results; the target recognition results represent: target operation actions corresponding to the target objects in the key frames, and target device states of the at least one target device; The evaluation module is used to evaluate the target operation and at least one of the target device states based on the preset corresponding relationship between the target operation action and at least one of the target device states and the evaluation criteria, and obtain the operation evaluation result corresponding to the target object.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the at least one processor executes the method as described in any one of claims 1 to 6 by executing the instructions stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store instructions, and when the instructions are executed, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising instructions, characterized in that The computer program product stores instructions, and when the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images