Production operation detection method, information display method, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a production operation detection method, an information display method, equipment, a storage medium and a program product, which are applied to the technical field of production and manufacturing, and comprise the following steps: obtaining a production operation video collected by video collection equipment; the production operation video is acquired by the video acquisition equipment by acquiring a production action executed by a target user; segmenting the production operation video to obtain a video clip corresponding to a target production process; identifying production action data in the video clip; and comparing the production action data with action reference data corresponding to the target production process to obtain a detection result. According to the scheme of the embodiment of the invention, the detection cost of the production action is reduced, and the detection efficiency and accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of production and manufacturing technology, and in particular, to a production operation detection method, an information display method, a device, a storage medium, and a program product. Background Art

[0002] In the manufacturing industry, strict implementation of the production process (i.e. a series of production processes and links in the manufacturing industry, usually consisting of a set of sequential production procedures) is the key to ensuring production efficiency and quality. Among them, the complete and accurate execution of production actions is an extremely important part of the smooth execution of the production process.

[0003] At present, manual inspection methods, such as manual inspection and manual sampling, are usually used to inspect the production actions of production operators. The inspection cost is high, the efficiency is low, and the inspection accuracy is not high. Summary of the invention

[0004] The embodiments of the present application provide a production operation detection method, an information display method, a device, a storage medium and a program product to solve the problems of high production action detection cost, low efficiency and low detection accuracy.

[0005] In a first aspect, the present application provides a production operation detection method, comprising:

[0006] Acquire a production operation video captured by a video capture device; the production operation video is obtained by capturing a production action performed by the video capture device on a target user;

[0007] Segmenting the production operation video to obtain a video segment corresponding to a target production process;

[0008] identifying production action data in the video clip;

[0009] The production action data is compared with the action reference data corresponding to the target production process to obtain a detection result.

[0010] In a second aspect, the present application provides an information display method, comprising:

[0011] Based on the first device identifier of the video acquisition device, a binding request is sent to the server; the binding request is used to request the server to establish an association relationship between the first device identifier and the user identifier of the target user, the association relationship is used to find the user identifier corresponding to the first device identifier of the video acquisition device, and the server sends a detection prompt message based on the user identifier; the detection prompt message is generated based on the detection result corresponding to the target production process; the detection result is obtained by comparing the production action data identified from the video clip corresponding to the target production process with the action reference data, and the video clip is obtained by the server by segmenting the production operation video collected by the video acquisition device;

[0012] Obtaining the detection prompt information sent by the server;

[0013] The detection prompt information is displayed in the user interface.

[0014] In a third aspect, the present application provides a computing device comprising a storage component and a processing component; the storage component stores one or more computer program instructions, the computer program instructions are called and executed by the processing component, and the processing component executes the one or more computer program instructions to implement the production operation detection method as described in the first aspect, or the information display method as described in the second aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a computer to implement the production operation detection method as described in the first aspect, or the information display method as described in the second aspect.

[0016] In a fifth aspect, the present application provides a computer program product storing a computer program, which, when executed by a computer, implements the production operation detection method as described in the first aspect, or the information display method as described in the second aspect.

[0017] In an embodiment of the present application, by identifying the production action data corresponding to the target production process from the production operation video and comparing it with the corresponding action benchmark data, automatic detection of the production actions of the production operators in the target production process is achieved without the need for manual detection, thereby reducing detection costs and improving detection efficiency and accuracy.

[0018] These and other aspects of the present application will become more clearly understood in the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flow chart of an embodiment of a production operation detection method provided by the present application is shown;

[0021] Figure 2 A flowchart of an embodiment of a method for generating action benchmark data provided by the present application is shown;

[0022] Figure 3 A flowchart of an embodiment of an information display method provided by the present application is shown;

[0023] Figure 4 A schematic diagram of a user interface provided in an example of the present application is shown;

[0024] Figure 5 A schematic structural diagram of an embodiment of a production operation detection device provided by the present application is shown;

[0025] Figure 6 A schematic structural diagram of an embodiment of an information display device provided by the present application is shown;

[0026] Figure 7 A system architecture diagram in a practical application is shown;

[0027] Figure 8 A schematic diagram of the structure of an embodiment of a computing device provided by the present application is shown. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0029] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0030] The technical solution of the embodiment of the present application can be applied to the production field, especially to the production operation detection in the production process. For ease of understanding, the technical terms that may appear in the embodiment of the present application are explained below accordingly:

[0031] Production process: a series of production processes and links in the manufacturing industry, usually composed of a set of sequential production processes, with clear series / parallel dependencies between the production processes;

[0032] Production process: a specific production step in the production process, usually consisting of a series of sequential production actions;

[0033] Production actions: technical actions performed by production operators, such as backstitching, notch alignment, etc., can be composed of a set of action elements, such as picking up, stacking, pushing / pulling, etc.;

[0034] Production equipment: equipment used by production operators during the execution of production actions, such as machine tools, blast furnaces, etc.

[0035] Production materials: items related to product production, such as materials, raw materials, auxiliary supplies, etc.

[0036] In the manufacturing industry, strict implementation of the production process is the key to ensuring production efficiency and quality. Among them, the complete and accurate execution of production actions is an extremely important part of the smooth execution of the production process.

[0037] At present, manual inspection methods, such as manual inspection and manual sampling, are usually used to inspect the production actions of production operators. The inspection cost is high, the efficiency is low, and the inspection accuracy is not high.

[0038] In order to solve the above technical problems, the inventors proposed the technical solution of the present application, that is, each workstation can deploy a video acquisition device to obtain the production operation video acquired by the video acquisition device; the production operation video is acquired by acquiring the production action performed by the video acquisition device on the target user; the video clip corresponding to the target production process is obtained by segmenting from the production operation video; the production action data in the video clip is identified; the production action data is compared with the action reference data corresponding to the target production process to obtain the detection result.

[0039] In this embodiment, by identifying the production action data corresponding to the target production process from the production operation video and comparing it with the corresponding action benchmark data, automatic detection of the production actions of the production operators in the target production process is achieved without the need for manual detection, thereby reducing detection costs and improving detection efficiency and accuracy.

[0040] And because the production operation video is obtained by capturing the production actions performed by the target user through the video capture device, for each production process or a specific production process performed by the target user, the corresponding video segment can be segmented from the production operation video, and the production action data in the video segment can be identified to perform automatic detection of the production action. The production operation detection method of this embodiment can effectively cover each production process performed by the target user, thereby improving the detection coverage.

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0042] The technical solution of the embodiment of the present application can be applied to a system architecture including a video acquisition device and a server, and the video acquisition device and the server are connected through a network. The network provides a medium for the communication link between the video acquisition device and the server. The network can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0043] The video acquisition device can send the acquired video to the server through the network. The video acquisition device may include a camera, a video recorder, etc. In an actual application, a corresponding video acquisition device can be configured for each workbench in the production workshop to acquire the production operation video of the production operator corresponding to the workbench.

[0044] Optionally, when the workbench is equipped with production equipment, the system may further include the production equipment, and the production equipment and the server may also be connected via a network, and the equipment operation data thereof may be sent to the server via the network. The production equipment may include machine tools, blast furnaces, and the like.

[0045] In addition, the system architecture may also involve a user end, and the user end and the server end may also be connected through a network, and the user end may interact with the server end through the network to send requests or receive information, etc. Optionally, the user end and the video acquisition device and / or the production device may also be connected through a network, and the user end may interact with the video acquisition device and / or the production device through the network to send or receive information, etc.

[0046] Among them, the user end can be a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5, the fifth edition of Hypertext Markup Language) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The user end can be deployed in an electronic device and needs to rely on the device to run or some apps in the device to run, etc. The electronic device can, for example, have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications can usually be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0047] The server may include a server that provides various services, such as a server that processes the captured video sent by the video capture device, or a server that processes the request sent by the user.

[0048] It should be noted that the server can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server of a distributed system, or a server combined with blockchain. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0049] It should be noted that the production operation detection method provided in the embodiment of the present application is generally executed by the server, and the corresponding production operation detection device is generally deployed in the server. However, in other embodiments of the present application, the user end may also have similar functions to the server, thereby executing the production operation detection method provided in the embodiment of the present application. In other embodiments, the production operation detection method provided in the embodiment of the present application may also be jointly executed by the user end and the server.

[0050] In addition, the information display method provided in the embodiments of the present application is generally executed by a user terminal, and the user terminal is provided with a user interface for information display.

[0051] It should be noted that the embodiments of the present application may involve the use of user data. In actual applications, user-specific personal data can be used in the scheme described herein within the scope permitted by applicable laws and regulations, subject to the requirements of applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0053] It should be noted that the technical solution of the embodiment of the present application is applicable to a network virtual environment. The users described generally refer to "virtual users". Real users can register a user account in the server through registration to obtain a user identity in the network environment.

[0054] like Figure 1 As shown, it is a flowchart of an embodiment of a production operation detection method provided by the present application. The method can be executed by the server and may include the following steps.

[0055] 101: Obtain a production operation video captured by a video capture device.

[0056] In this embodiment, the production operation video may be obtained by capturing the production action performed by the target user by a video capture device. Optionally, the production action performed by the target user may be a production action performed by the target user in a production workshop based on production equipment and / or production materials.

[0057] In an actual application, a production workshop may be provided with a plurality of video acquisition devices, each of which may correspond to a production operator. When production equipment is deployed in the production workshop, that is, each of which may correspond to a production equipment, the method may further include:

[0058] Obtain a binding request sent by the user end, which may include a first device identifier corresponding to the video acquisition device, and establish an association relationship between the first device identifier and a user identifier of a target user corresponding to the user end. The binding request may be generated by the user end in response to a scanning operation on a medium carrying the first device identifier on the video acquisition device, for example, and this application does not limit this. The first device identifier may uniquely identify the corresponding video acquisition device.

[0059] On this basis, after obtaining the production operation video captured by the video capture device, the method may further include:

[0060] According to the first device identifier of the video acquisition device, an association relationship is searched to determine a corresponding user identifier.

[0061] Thus, the server can determine the target user corresponding to the production operation video, and can detect the production action of the target user based on the production operation video. The target user can be any production operator.

[0062] 102: Segment the production operation video to obtain a video segment corresponding to a target production process.

[0063] As an optional implementation method, the target production process may refer to a specific production process with a detection requirement. In this case, a video clip of the target production process may be obtained by segmenting from the production operation video. Specifically, the corresponding video clip may be segmented from the production operation video according to the production start and end time or production start and end action of the target production process. The production start and end time may be pre-set, and the production start and end action may be identified from the video. The specific identification method will be described in the subsequent embodiments.

[0064] As another optional implementation, the target production process may refer to any production process. In this case, video clips corresponding to multiple production processes may be obtained by segmenting the production operation video, and each production process may be used as the target process to obtain video clips corresponding to the target production process. Among them, video clips corresponding to multiple production processes may be segmented according to the production start and end time or production start and end action of each production process.

[0065] In addition, the video segmentation can also be performed in combination with the equipment operation data of the production equipment, which will be explained in the subsequent embodiments.

[0066] It should be noted that in the actual production process, the target user may be responsible for executing only one production process in the production process for the production material. In this case, the production process is the target production process. Of course, the target user may also execute the complete production process or multiple production processes in the production process for the production material. In this case, the target production process can be determined according to the above implementation method.

[0067] 103: Identify production action data in the video clip.

[0068] The production action data may include the action sequence, the time interval between adjacent production actions in the action sequence, the body movement amplitude and movement range of each production action in the action sequence, etc., which can be determined according to the actual production scenario. The recognition of production action data can be achieved by using traditional computer vision algorithms, deep learning-based recognition algorithms, and other methods.

[0069] 104: Compare the production action data with the action reference data corresponding to the target production process to obtain a detection result.

[0070] Among them, the action benchmark data can be data pre-generated or acquired according to the requirements of the target production process for the technical actions of production operators, process specifications and quality requirements, the specifications and parameter requirements of the production equipment and fixtures used, the material and shape specifications of the production materials used, etc. For example, it can include the action benchmark of each production action, such as action elements, action positions, etc., the execution order benchmark of multiple production actions, etc.

[0071] By comparing the production action data with the action benchmark data, one or more comparison results can be obtained. For example, each production action can be compared with the corresponding action benchmark to obtain the corresponding comparison result. For example, the production action is to push forward and pull backward, and the corresponding action benchmark is to pull forward and push backward. The comparison result obtained can be: the production action is wrong. Based on one or more comparison results, a detection result can be obtained. Optionally, one or more comparison results can be used directly as the detection result, and the scores corresponding to one or more comparison results can be calculated, and weighted processing is performed in combination with the corresponding weight coefficients to obtain the final detection score as the detection result. It can be set according to actual needs.

[0072] Optionally, when comparing the production action data with the action benchmark data, the comparison can be performed according to predefined detection rules to obtain the detection results. The detection rules can include comparison rules, scoring rules, weight coefficient setting rules, etc., which can be set according to actual needs.

[0073] In this embodiment, by identifying the production action data corresponding to the target production process from the production operation video and comparing it with the corresponding action benchmark data, automatic detection of the production actions of the production operators in the target production process is achieved. There is no need for manual detection, which reduces the detection cost and improves the detection efficiency and accuracy. In addition, since the production operation video is obtained by capturing the production actions performed by the target user through a video acquisition device, for each production process or a specific production process performed by the target user, the corresponding video clip can be segmented from the production operation video, and the production action data in the video clip can be identified to perform automatic detection of the production action. The production operation detection method of this embodiment can effectively cover each production process performed by the target user, thereby improving the detection coverage.

[0074] The process of video segmentation is described below, which can be implemented in multiple ways.

[0075] As an optional implementation, it can be implemented in combination with the equipment operation data of the production equipment. In this implementation, the production operation video can be obtained by the video acquisition device capturing the production actions performed by the target user based on the production equipment. Therefore, the above method can also include:

[0076] Obtain equipment operation data of production equipment.

[0077] At this time, the method of segmenting the production operation video to obtain the video segment corresponding to the target production process may include:

[0078] Time-align equipment operation data with production operation videos;

[0079] According to the equipment operation data corresponding to the target production process, the production start time and production end time corresponding to the target production process are determined, and according to the production start time and production end time, the video clip corresponding to the target production process is obtained from the production operation video segmentation.

[0080] Among them, the equipment operation data may refer to data such as temperature, pressure, speed, load, current, etc. generated by the operation of the production equipment during the production process. It is understandable that in the actual production process, when executing different production processes, the equipment operation data of the production equipment is usually different. Taking the two different production processes of sewing buttons and sewing sleeves as examples, when executing the production process of sewing buttons, the working frequency and moving trajectory of the sewing machine needle are different from those when executing the production process of sewing sleeves, and the corresponding equipment operation data are also different. In addition, when the same production process is executed cyclically, the equipment operation data of the production equipment can usually show periodic fluctuation characteristics. For example, when the production process of sewing buttons is executed cyclically, when sewing each button, the working frequency and moving trajectory of the sewing machine needle are usually the same, and the corresponding equipment operation data are also the same, and they are different from the equipment operation data corresponding to the sewing interval of two buttons, thereby showing periodic fluctuation characteristics. Therefore, whether executing different production processes or cyclically executing the same production process, the production start and end times corresponding to the target production process can be determined according to the equipment operation data of the production equipment.

[0081] Based on this, the video clip of the target production process can be segmented from the production operation video in combination with the production start and end times corresponding to the target production process. Specifically, the equipment operation data and the production operation video can be time-aligned, and the production start and end times corresponding to the target production process can be determined based on the equipment operation data, and the video clip between the production start and end times can be segmented from the production operation video, which is the video clip corresponding to the target production process.

[0082] In an actual application, a production workshop may be provided with multiple production equipment, and the binding request obtained by the server may also include the second device identification of the production equipment, and establish an association relationship between the second device identification and the user identification of the target user corresponding to the user end. The binding request may be generated by the user end in response to a scanning operation of a medium carrying the second device identification on the production equipment, for example, and this application does not limit this. The second device identification may uniquely identify the corresponding production equipment.

[0083] In this embodiment, by combining the equipment operation data of the production equipment and performing time alignment processing with the production operation video, the production start and end times of the target production process are determined based on the equipment operation data, and then the video clips between the production start time and the production end time are obtained from the production operation video, thereby improving the accuracy of video segmentation.

[0084] As another optional implementation method of video segmentation, it can be implemented directly based on the production operation video. In this implementation method, the method of segmenting the production operation video to obtain the video clip corresponding to the target production process may include:

[0085] Identify the starting production action and the ending production action corresponding to the target production process from the production operation video;

[0086] According to the starting production action and the ending production action, the video clips corresponding to the target production process are obtained from the production operation video segmentation.

[0087] In the actual production process, the starting production action and the ending production action of different production processes are usually different. For example, the starting production action of the production process of sewing buttons is usually to pick up the buttons to be sewn, and the ending production action is usually to raise the hand away from the buttons and clothes sewn together, while the starting production action of the production process of sewing sleeves is usually to pick up the sleeves to be sewn, and the ending production action is usually to raise the hand away from the pressed sleeves. Therefore, when executing different production processes, the video clips corresponding to the target production process can be segmented according to the starting production action and the ending production action. When the same production process is executed in a loop, the video clips corresponding to multiple production processes can be segmented according to the periodic starting production action and the ending production action. Among them, the recognition of the starting production action and the ending production action can be achieved using traditional computer vision algorithms, deep learning-based recognition algorithms, etc., or can also be achieved using the action recognition model recorded in the subsequent embodiments.

[0088] In this embodiment, by directly identifying the starting production action and the ending production action of the target production process, the video clips of the target production process are obtained by segmenting the production operation video according to the starting and ending production actions, thereby improving the video segmentation efficiency.

[0089] For the segmented video clips of the target production process, the production action data in the clips can be identified. The implementation of identifying the production action data in the video clips is described below.

[0090] In order to improve the recognition accuracy of production action data, an action recognition model may be used. Therefore, in some embodiments, the method of recognizing production action data in a video clip may include:

[0091] An action recognition model is used to recognize action sequences in video clips and time intervals between adjacent production actions in the action sequences; wherein the action recognition model can be obtained by training based on production sample videos and training labels.

[0092] Optionally, the above method may further include:

[0093] Obtaining production sample videos and training labels; the training labels may include sample motion data formed by the interaction between limbs and production materials and / or production equipment;

[0094] The action recognition model is trained using the production sample videos and training labels to obtain the model parameters of the action recognition model.

[0095] Among them, the production sample video can be obtained by collecting the production actions performed on the sample user, or the historical production operation video stored in the database can be obtained. The training label can be the sample action data marked in the production sample video. Taking sewing manufacturing as an example, the user's limbs can include the user's hands, the production materials can include the clothes to be sewn, and the production equipment can include a sewing machine. The sample action data formed by the interaction between the user's limbs and the production materials can include, for example, picking up the clothes to be sewn with the hands, unfolding the clothes to be sewn with the hands, etc., the sample action data formed by the interaction between the user's limbs and the production equipment can include, for example, pressing the needle lifting rod knob of the sewing machine with the hands to lift or drop the needle, etc., the sample action data formed by the interaction between the user's limbs and the production materials and production equipment can include, for example, pressing the needle lifting rod knob of the sewing machine with one hand to lift the needle, picking up the clothes to be sewn with one hand and placing them under the needle, etc.

[0096] The production sample video and the above training labels are input into the action recognition model, and the loss function is calculated based on the output data and the above training labels. The model parameters of the action recognition model are adjusted until the loss function meets the requirements to obtain the corresponding model parameters.

[0097] Optionally, the action recognition model can identify the action sequence and determine the time interval between two production actions based on the end time of the previous production action and the start time of the next production action in the action sequence. Alternatively, the time interval between two adjacent production actions can be determined from the production operation video in combination with the identified action sequence and the start and end times of each production action.

[0098] In this embodiment, the action recognition model is trained by using production sample videos and training labels. Since the training labels include sample action data consisting of interactions between limbs and production materials and / or production equipment, the accuracy of the action recognition model can be improved, thereby improving the accuracy of identifying production action data from video clips, thereby achieving accurate detection of production actions.

[0099] After the production action data is identified, it can be detected in combination with the corresponding action reference data, and the detection process is described below. In some embodiments, the action reference data may include action sequence reference data and interval reference data between adjacent production actions.

[0100] Specifically, the method of comparing the production action data with the action reference data corresponding to the target production process to obtain the detection result may include:

[0101] Comparing the action sequence with corresponding action sequence benchmark data to obtain sequence comparison data;

[0102] Compare the time interval between two adjacent production actions with the corresponding interval reference data to obtain time comparison data;

[0103] Based on the sequence comparison data and the time comparison data, a detection result is obtained.

[0104] Among them, the action sequence benchmark data may include the action standard sequence and the execution order error data or execution number error data of the production actions in the action standard sequence. The action standard sequence may include a series of ordered production actions corresponding to the complete and accurate execution of the target production process, and the execution order, execution number, etc. of each production action are determined. The execution order error data may refer to the execution order of one or more production actions that are allowed to be changed in the actual production process, and the execution number error data may refer to the execution number of one or more production actions that are allowed to be changed in the actual production process. For example, the action sequence benchmark data includes: production action A-(production action A that can be selected to be executed again)-production action B-production action C, wherein the action standard sequence includes: production action A-production action B-production action C, and the execution number error data includes: production action A is allowed to be executed twice.

[0105] The interval reference data may include the time standard interval and interval error data between adjacent production actions in the action standard sequence. The time standard interval is fixed, and the interval error data may refer to the time interval that is allowed to be exceeded or shortened in the actual production process. For example, the interval reference data between production action A and production action B is 4 seconds to 7 seconds, where the time standard interval is 5 seconds, and the interval error data includes shortening by 1 second or extending by 2 seconds.

[0106] When performing production operation detection, the identified action sequence can be compared with the above-mentioned action sequence benchmark data to obtain sequence comparison data, which can be used to characterize the degree of completion of the production action. For example, the identified action sequence includes production action A-production action B, and the obtained sequence comparison data can be: production action C is missing. For another example, the identified action sequence includes production action A-production action A-production action C-production action B, and the obtained sequence comparison data can be: the execution order of production actions B and C is wrong. The above sequence comparison data can all indicate that the degree of completion of the production action of the target production process is not high.

[0107] By comparing the time interval between two adjacent production actions with the corresponding interval reference data, time comparison data can be obtained, and the time comparison data can be used to characterize the rationality of the time interval between two adjacent production actions. For example, the interval reference data between production actions A and B is 7 seconds, and the identified time interval is 3 seconds. The obtained time series data may be: the time interval between production actions A and B is short, production action A ends late, or production action B starts early, etc. Correspondingly, it can be shown that the execution end time of production action A is unreasonable, or the execution start time of production action B is unreasonable, etc.

[0108] Based on the sequence comparison data and time comparison data obtained above, a test result can be obtained. For example, the sequence comparison data and the time comparison data can be directly used as the test result, or the scores of the sequence comparison data and the time comparison data can be calculated respectively, and weighted processing is performed according to the corresponding weight coefficient to obtain the final test score as the test result.

[0109] In this embodiment, by comparing the identified action sequence with the action sequence benchmark data to obtain the sequence comparison data, and comparing the time interval between adjacent production actions with the interval benchmark data to obtain the time comparison data, it is possible to detect the degree of completion of the production action and the time rationality, thereby improving the comprehensiveness of the production action detection. At the same time, the action sequence benchmark data includes the action standard sequence and also sets the execution order or execution number error data for the production action, and the interval benchmark data includes the time standard interval and also sets the interval error data, thereby improving the rationality of the action sequence benchmark data setting, and being able to perform reasonable production action detection without affecting the degree of completion of the production action and the time rationality, taking into account factors such as the operating habits of different production operators.

[0110] In some embodiments, the action benchmark data may also include production action error data and predetermined type actions; the predetermined type actions may include redundant actions and / or risky actions.

[0111] Specifically, comparing the production action data with the action reference data corresponding to the target production process to obtain the detection result may also include:

[0112] Compare the production actions in the action sequence with the production action error data to obtain action comparison data;

[0113] Identify whether the action sequence includes a predetermined type of action, and obtain an action recognition result;

[0114] A detection result is obtained based on the sequence comparison data, the time comparison data, the motion comparison data and the motion recognition result.

[0115] Among them, the production action error data may include action element errors such as the amplitude of limb movements, action position errors such as the movement range of production materials, the area range of interaction between limbs and production materials or production equipment, etc. Redundant actions may refer to unnecessary actions that fail to directly generate production value during the production process, and may include unnecessary limb movements, unnecessary interactions between limbs and production materials and / or production equipment, etc. Generally, the more redundant actions, the lower the production efficiency. Risky actions may refer to actions that may cause damage to the human body, production materials and / or production equipment, such as actions in which a user touches the tip of a needle in a sewing machine with his fingers, which may cause damage to his fingers.

[0116] When performing production operation detection, the identified production action can be compared with the production action error data to obtain action comparison data, which can be used to characterize the accuracy of the production action. For example, the error data of production action A includes that the movement range of the needle in the sewing machine does not exceed 5 cm, and the movement range of the needle in the identified production action A is 6 cm. The obtained action comparison data may be: the needle movement distance is too long. In addition, it can be detected and identified whether the above-mentioned predetermined type of action exists in the action sequence to obtain the action recognition result. The action recognition result may include: the existence of redundant actions and the number of redundant actions and / or the existence of risky actions and the number of risky actions, and the absence of redundant actions and risky actions.

[0117] The final detection result can be obtained by combining the sequence comparison data, time comparison data, action comparison data and action recognition results in the aforementioned embodiments. For example, the sequence comparison data, time comparison data, action comparison data and action recognition results can be directly used as the detection result, or the scores of the sequence comparison data, time comparison data, action comparison data and action recognition results can be calculated respectively, and weighted according to the corresponding weight coefficients to obtain the final detection score as the detection result.

[0118] In this embodiment, by combining action comparison data and action recognition results, it is possible to detect multiple aspects such as the degree of completion of the production action, time rationality, action accuracy, production efficiency and action safety, thereby further improving the comprehensiveness of production action detection.

[0119] In practical applications, the above action benchmark data can also be identified and generated using an action recognition model. Figure 2 As shown, it is a flowchart of an embodiment of a method for generating action benchmark data provided by the present application. The method can also be executed by the server and may include the following steps.

[0120] 201: Obtain the standard action video corresponding to the target production process.

[0121] Among them, the standard action video can be a video pre-recorded by a designated production operator performing a series of sequential production actions in accordance with the target production process's requirements for the production operator's technical actions, process specifications and quality requirements, the specifications and parameter requirements of the production equipment and fixtures used, the material and shape specifications of the production materials used, etc. Optionally, the standard action video corresponding to the target production process can also be obtained from a database, wherein the database can store standard action videos corresponding to multiple production processes in the corresponding manufacturing industry, and this application does not limit this.

[0122] 202: Using an action recognition model to identify a standard action sequence in a standard action video and a standard time interval between adjacent production actions in the standard action sequence.

[0123] 203: Generate action sequence benchmark data based on the action standard sequence and the execution order error data and / or execution number error data of the production action.

[0124] 204: Generate interval benchmark data based on the time standard interval and interval error data between adjacent production actions in the action standard sequence.

[0125] 205: Generate motion error data based on motion element errors and / or motion position errors of the production motion in the motion standard sequence.

[0126] 206: Determine a predetermined type of action in a standard sequence of actions.

[0127] 207: Generate action reference data including action sequence reference data, interval reference data between adjacent production actions, action error data and action reference data of predetermined types of actions.

[0128] In this embodiment, by acquiring standard action videos and using action recognition models to identify the standard action sequence and the standard time intervals between adjacent production actions, and setting execution sequence error data and execution number error data, interval error data, action element errors and / or action position errors and predetermined types of actions, the generation of action benchmark data is achieved, thereby improving the rationality of the action benchmark data.

[0129] Of course, in actual applications, it is also possible to directly generate a standard action sequence and the standard time intervals between adjacent production actions in the standard action sequence based on the target production process's requirements for the production operators' technical actions, process specifications and quality requirements, the specifications and parameter requirements of the production equipment and fixtures used, the material and shape specifications of the production materials used, etc., and reasonably set the corresponding error data to generate action benchmark data.

[0130] Optionally, a corresponding standard action video may also be generated based on the action benchmark data.

[0131] In some embodiments, the above method may further include:

[0132] Generate detection prompt information based on multiple detection results corresponding to multiple video clips corresponding to the target production process within a preset time range;

[0133] Notify the target user of the detection prompt information.

[0134] Among them, the detection prompt information may include detection summary results for multiple detection results, action adjustment prompt information, action guidance suggestions, etc. Specifically, the action adjustment prompt information and action guidance suggestions may be generated based on multiple detection results. For example, the detection summary result may be: production action A is missing for five consecutive production processes, and the action adjustment prompt information may be: add production action A; for another example, the detection summary result may be: the body movement amplitude of production action B is too large, and the action adjustment prompt information or action guidance suggestions may be: reduce the body movement amplitude of production action B, etc.

[0135] Optionally, the detection prompt information may be generated based on multiple detection results corresponding to the same production process within a preset time range, or may be generated based on detection results corresponding to different production processes within a preset time range.

[0136] Optionally, when the detection result is obtained, the user identification of the target user may be bound to the detection result, so that detection prompt information corresponding to the detection result is sent to the corresponding user terminal based on the user identification.

[0137] By generating detection prompt information based on multiple detection results within a preset time range, and passing it to the target user, the target user can clearly understand his or her own production execution status, and make corresponding production operation adjustments according to the corresponding action adjustment prompt information or guidance suggestions, thereby improving the accuracy of production operations.

[0138] Optionally, the above method may further include:

[0139] When it is determined based on the detection results that there are risky actions in the action sequence, risk warning information is generated and notified to the target user. The risk warning information may include existing risky actions, action guidance suggestions, etc., to prompt the target user to promptly clarify the risky actions they are performing and make corresponding adjustments to improve the safety of production operations.

[0140] Optionally, the server may also evaluate the target user based on the detection result of the target user, for example, determine the user level or user rights, etc., to incentivize users with high operation accuracy, etc.

[0141] Optionally, the above method may further include:

[0142] The standard action video is sent to the user terminal corresponding to the target user to prompt the target user to adjust the action according to the standard action video, etc., so as to further improve the accuracy of production operation.

[0143] The above one or more embodiments are written from the perspective of execution on the server side. The following is an explanation of the technical solution of the present application from the perspective of execution on the user side. Figure 3 As shown, it is a flowchart of an embodiment of an information display method provided by the present application. The method can be executed by a user terminal and may include the following steps.

[0144] 301: Based on the first device identifier of the video acquisition device, a binding request is sent to the server.

[0145] Among them, the binding request can be used to request the server to establish an association relationship between the first device identifier and the user identifier of the target user. The established association relationship can be used to find the user identifier corresponding to the first device identifier of the video acquisition device, and the server sends a detection prompt message based on the user identifier. The detection prompt message can be generated based on the detection result corresponding to the target production process. The detection result is obtained by the server based on the comparison of the production action data identified from the video clip corresponding to the target production process and the action reference data. The video clip is obtained by the server by segmenting the production operation video collected by the video acquisition device. The specific process of generating the detection result is described in Figure 1 The corresponding description has been made in the illustrated embodiment and will not be repeated here.

[0146] 302: Get the detection prompt information sent by the server.

[0147] Among them, the detection prompt information can be generated by the server based on multiple detection results corresponding to multiple video clips corresponding to the target production process within a preset time range, and can include detection summary results, action adjustment prompt information, action guidance suggestion information, etc.

[0148] Optionally, risk warning information sent by the server may also be obtained. The risk warning information may be generated by the server when it is determined that there is a risky action in the action sequence based on the detection result.

[0149] 303: Display detection prompt information in the user interface.

[0150] Optionally, in order to improve the display effect, the production operation video sent by the server may be obtained and played in the user interface.

[0151] In this embodiment, by identifying the production action data corresponding to the target production process from the production operation video and comparing it with the corresponding action benchmark data, automatic detection of the production actions of the production operators in the target production process is achieved. There is no need for manual detection, which reduces the detection cost and improves the detection efficiency and accuracy. In addition, since the production operation video is obtained by capturing the production actions performed by the target user through a video acquisition device, for each production process or a specific production process performed by the target user, the corresponding video clip can be segmented from the production operation video, and the production action data in the video clip can be identified to perform automatic detection of the production action. The production operation detection method of this embodiment can effectively cover each production process performed by the target user, thereby improving the detection coverage.

[0152] In some embodiments, the above method may further include:

[0153] Obtain the standard action video sent by the server and play the standard action video to facilitate the target user to learn and adjust the production action to improve the accuracy of production operation.

[0154] In some embodiments, the above method may further include:

[0155] In response to a scanning operation on a medium carrying a first device identification in a video acquisition device, a first device identification of the video acquisition device is acquired.

[0156] The medium carrying the first device identification in the video acquisition device may include, for example, a QR code. In response to a scanning operation by a target user, the first device identification of the video acquisition device may be obtained, and a binding request may be sent to the server.

[0157] For ease of understanding, Figure 4 A schematic diagram of a user interface is shown. It should be noted that: Figure 4 This is only an exemplary description, and the interface display method of this application is not limited to this. In actual applications, different interface styles can be designed in combination with different needs. Figure 4 As shown, the user interface 40 may display a video screen of a production operation video, and detection prompt information 401 may be superimposed on the video screen. The detection prompt information may include, for example, detection summary results such as a detection score of "86 points" and action adjustment prompt information such as "5 consecutive pieces of missing [alignment] actions".

[0158] The total detection score may be, for example, the average value of the detection scores of multiple detection results.

[0159] In addition, standard prompt information 402 may be displayed in the user interface 40. The standard prompt information may be, for example, a text message of "view explanation". In response to the triggering operation of the standard prompt information 402, a standard action video may be played, and the corresponding video screen of the standard action video may be displayed in the user interface to facilitate the target user to learn, etc. In addition, according to the playing time of the standard action video, corresponding time prompt information may be displayed in the user interface.

[0160] like Figure 5 As shown, it is a structural schematic diagram of an embodiment of a production operation detection device provided by the present application, and the device may include the following modules.

[0161] The first acquisition module 501 is used to acquire a production operation video acquired by a video acquisition device; the production operation video is acquired by the video acquisition device acquiring a production action performed by a target user;

[0162] A segmentation module 502 is used to segment the production operation video to obtain a video segment corresponding to a target production process;

[0163] A first recognition module 403, used to recognize production action data in the video clip;

[0164] The detection module 504 is used to compare the production action data with the action reference data corresponding to the target production process to obtain a detection result.

[0165] In some embodiments, the production operation video is obtained by capturing the production action performed by the target user based on the production equipment using a video capture device; the above-mentioned device may also include:

[0166] The second acquisition module is used to acquire the equipment operation data of the production equipment;

[0167] The segmentation module 502 can be specifically used to time-align the equipment operation data with the production operation video; determine the production start time and production end time corresponding to the target production process based on the equipment operation data corresponding to the target production process, and obtain the video clip corresponding to the target production process from the production operation video according to the production start time and production end time.

[0168] In some embodiments, the segmentation module 502 can be specifically used to identify the starting production action and the ending production action corresponding to the target production process from the production operation video; according to the starting production action and the ending production action, the video clip corresponding to the target production process is obtained from the production operation video segmentation.

[0169] In some embodiments, the first recognition module 503 can be specifically used to use an action recognition model to recognize action sequences in video clips and time intervals between adjacent production actions in the action sequence; wherein the action recognition model is trained based on production sample videos and training labels.

[0170] In some embodiments, the action benchmark data may include action sequence benchmark data and interval benchmark data between adjacent production actions;

[0171] The detection module 504 can be specifically used to compare the action sequence with the corresponding action sequence benchmark data to obtain sequence comparison data; compare the time interval between two adjacent production actions with the corresponding interval benchmark data to obtain time comparison data; and obtain the detection result based on the sequence comparison data and the time comparison data.

[0172] In some embodiments, the action benchmark data may also include production action error data and predetermined type actions; the predetermined type actions include redundant actions and / or risky actions;

[0173] The detection module 504 can be specifically used to compare the production action in the action sequence with the production action error data to obtain action comparison data; identify whether the action sequence includes a predetermined type of action to obtain an action recognition result; and obtain a detection result based on the sequence comparison data, time comparison data, action comparison data and action recognition result.

[0174] In some embodiments, the above apparatus may further include:

[0175] The third acquisition module is used to acquire production sample videos and training labels; the training labels include sample action data formed by the interaction between the limbs and the production materials and / or production equipment;

[0176] The training module is used to train the action recognition model using the production sample videos and training labels to obtain the model parameters of the action recognition model.

[0177] In some embodiments, the above apparatus may further include:

[0178] A fourth acquisition module is used to acquire a standard action video corresponding to a target production process;

[0179] A second recognition module is used to recognize a standard action sequence in a standard action video and a standard time interval between adjacent production actions in the standard action sequence by using an action recognition model;

[0180] The first generating module is used to generate action sequence benchmark data based on the execution order error data and execution number error data of the production action in the action standard sequence; generate interval benchmark data based on the time standard interval and interval error data between adjacent production actions in the action standard sequence; generate action error data based on the action element error and / or action position error of the production action in the action standard sequence;

[0181] The determination module is used to determine a predetermined type of action in a standard sequence of actions.

[0182] In some embodiments, the above apparatus may further include:

[0183] A second generating module, used to generate detection prompt information based on multiple detection results within a preset time range;

[0184] The first notification module is used to notify the target user of the detection prompt information.

[0185] In some embodiments, the above apparatus may further include:

[0186] A third generating module is used to generate risk warning information when it is determined that there is a risk action in the action sequence based on the detection result;

[0187] The second notification module is used to notify the target user of the risk warning information.

[0188] In some embodiments, the above apparatus may further include:

[0189] A fourth acquisition module is used to acquire a binding request sent by the user terminal; the binding request includes a first device identifier corresponding to the video acquisition device;

[0190] A first association module, used to establish an association relationship between the first device identifier and a user identifier of a target user corresponding to the user terminal;

[0191] The device may also include:

[0192] The second association module is used to find the association relationship and determine the corresponding user identification according to the first device identification of the video acquisition device; and establish the corresponding relationship between the user identification and the detection result.

[0193] In some embodiments, the above apparatus may further include:

[0194] The first sending module is used to send detection prompt information corresponding to the detection result to the corresponding user terminal based on the user identifier.

[0195] In some embodiments, the above apparatus may further include:

[0196] The second sending module is used to send the standard action video to the user terminal corresponding to the target user.

[0197] In some embodiments, the segmentation module 502 can be specifically used to segment the production operation video to obtain video segments corresponding to multiple production processes; any production process is used as a target production process to obtain a video segment corresponding to the target production process.

[0198] like Figure 6 As shown, it is a structural schematic diagram of an embodiment of an information display device provided by the present application, and the device may include the following modules.

[0199] The third sending module 601 is used to send a binding request to the server based on the first device identifier of the video acquisition device; the binding request is used to request the server to establish an association relationship between the first device identifier and the user identifier of the target user, and the association relationship is used to find the user identifier corresponding to the first device identifier of the video acquisition device, and the server sends the detection prompt information based on the user identifier; the detection prompt information is generated based on the detection result corresponding to the target production process; the detection result is obtained by comparing the production action data identified from the video clip corresponding to the target production process with the action reference data, and the video clip is obtained by the server by segmenting the production operation video collected by the video acquisition device;

[0200] A fifth acquisition module 602 is used to acquire detection prompt information sent by the server, where the detection prompt information is generated by the server based on multiple detection results within a preset time range;

[0201] The first display module 603 is used to display detection prompt information in the user interface.

[0202] In some embodiments, the above apparatus may further include:

[0203] The playback module is used to obtain the standard action video sent by the server and play the standard action video.

[0204] In some embodiments, the above apparatus may further include:

[0205] The sixth acquisition module is used to acquire the first device identification of the video acquisition device in response to a scanning operation on the medium carrying the first device identification in the video acquisition device.

[0206] For ease of understanding, the following Figure 7 The system architecture diagram shown in the figure illustrates the technical solution of this application. Figure 7 As shown, the system structure may include a server 701 , at least one client 702 , and corresponding video acquisition equipment 703 and production equipment 704 .

[0207] It should be noted that Figure 7 The presented server 701, user end 702, video acquisition device 703, and production equipment 704 are only exemplary and do not limit their implementation forms. For easy viewing, the user end diagram is represented by a device image, such as the mobile phone image in the figure, the video acquisition device diagram is also represented by a device image, such as the camera image in the figure, and the production equipment is the sewing machine image in the figure. It can be understood that the number of video acquisition devices 703 and production equipment 704 is only schematic. In actual applications, each workbench in the production workshop can be equipped with a corresponding video acquisition device 703 and production equipment 704.

[0208] During the actual production process, the target user enters the workbench and starts the production equipment 704 to start executing production actions for production materials and / or production equipment according to the production process.

[0209] The video acquisition device 703 senses the start of the production equipment 704 or detects that the target user enters the workbench, and can start collecting the production operation video of the target user performing production actions based on production materials and / or production equipment, and can upload it to the server 701 in real time.

[0210] Optionally, the user terminal 702 may respond to a scanning operation triggered by the target user for a medium carrying a first device identifier in the video acquisition device 703, obtain the first device identifier of the video acquisition device 703, and respond to a scanning operation triggered by the target user for a medium carrying a second device identifier in the production device 707, obtain the second device identifier of the production device 704, and send a binding request to the server terminal 701. The server terminal 701 may establish an association relationship between the first device identifier, the second device identifier, and the user identifier of the target user corresponding to the user terminal 702.

[0211] The server 701 can identify the production action data corresponding to the target production process from the production operation video, and compare it with the corresponding action benchmark data to obtain the detection result, thereby realizing automatic detection of the production action of the target user in the target production process, without the need for manual detection, reducing the detection cost, and improving the detection efficiency and accuracy. Moreover, since the production operation video is obtained by collecting the production actions performed by the target user, for each production process or a specific production process performed by the target user, the corresponding video segment can be segmented from the production operation video, and the production action data in the video segment can be identified to perform automatic detection of the production action. The production operation detection method of the embodiment of the present application can effectively cover each production process performed by the target user, thereby improving the detection coverage.

[0212] Optionally, the production operation video can be obtained by collecting the production actions performed by the target user based on the production equipment. The server 701 can also obtain the equipment operation data of the production equipment 704, perform time alignment processing on the equipment operation data and the production operation video, determine the production start time and production end time corresponding to the target production process according to the equipment operation data corresponding to the target production process, and obtain the video segment corresponding to the target production process from the production operation video according to the production start time and production end time. By combining the equipment operation data of the production equipment, performing time alignment processing on it and the production operation video, determining the production start and end time of the target production process according to the equipment operation data, and then obtaining the video segment between the production start time and production end time from the production operation video, the accuracy of video segmentation is improved.

[0213] Optionally, the server 701 may also directly identify the starting production action and the ending production action of the target production process, and obtain video clips of the target production process from the production operation video according to the starting and ending production actions, so as to improve the video segmentation efficiency.

[0214] Optionally, in order to improve the recognition accuracy of production action data, the server 701 can also use an action recognition model to recognize action sequences in video clips and the time intervals between adjacent production actions in the action sequence. The action recognition model can be obtained based on production sample videos and training labels. Prior to this, the server 701 can also obtain production sample videos and training labels. The training labels can include sample action data formed by the interaction between limbs and production materials and / or production equipment, and use the production sample videos and training labels to train the action recognition model to obtain the model parameters of the action recognition model. This improves the accuracy of the action recognition model, and then improves the accuracy of identifying production action data from video clips, and achieves accurate detection of production actions.

[0215] Optionally, the action benchmark data may include action sequence benchmark data and interval benchmark data between adjacent production actions. The server 701 may obtain sequence comparison data by comparing the identified action sequence with the action sequence benchmark data, and obtain time comparison data by comparing the time interval between adjacent production actions with the interval benchmark data, thereby realizing detection from two aspects of the degree of completion of the production action and the rationality of time, thereby improving the comprehensiveness of the production action detection. At the same time, the action sequence benchmark data includes the action standard sequence and also sets the execution order or execution number error data for the production action, and the interval benchmark data includes the time standard interval and also sets the interval error data, thereby improving the rationality of the action sequence benchmark data setting, and being able to perform reasonable production action detection by considering factors such as the operating habits of different production operators without affecting the degree of completion of the production action and the rationality of time.

[0216] Optionally, the action benchmark data may also include production action error data and predetermined type actions; the predetermined type actions may include redundant actions and / or risky actions. The server 701 may also compare the production actions in the action sequence with the production action error data to obtain action comparison data, and identify whether the action sequence includes predetermined type actions to obtain action recognition results. By combining the action comparison data and the action recognition results, it is possible to detect the degree of completion of the production action, time rationality, action accuracy, production efficiency, action safety and other aspects, thereby further improving the comprehensiveness of production action detection.

[0217] Optionally, the above-mentioned action benchmark data can also be identified and generated using an action recognition model. Specifically, the server 701 can obtain a standard action video corresponding to the target production process, which can be a video pre-recorded by a designated production operator according to the target production process requirements for the production operator's technical actions, process specifications and quality requirements, the specifications and parameter requirements of the production equipment and fixtures used, the material and shape specifications of the production materials used, etc. to perform a series of sequential production actions, or directly obtained from a database. The server 701 can use the action recognition model to identify the standard sequence of actions and the time standard interval between adjacent production actions, and set the execution sequence error data and the execution number error data, interval error data, action element error and / or action position error and predetermined type of action, thereby realizing the generation of action benchmark data and improving the rationality of the action benchmark data.

[0218] Optionally, the server 701 can also generate detection prompt information based on multiple detection results within a preset time range, and pass it to the target user so that the target user can clearly understand the execution of his own production, and make corresponding production operation adjustments according to the corresponding action adjustment prompt information or guidance suggestions, so as to improve the accuracy of production operations. And, based on the detection results, when it is determined that there are risky actions in the action sequence, risk warning information is generated and the risk warning information is notified to the target user. Among them, the risk warning information may include existing risky actions, action guidance suggestions, etc., to prompt the target user to promptly clarify the risky actions performed by himself and make corresponding adjustments, so as to improve the safety of production operations.

[0219] Optionally, the server may also send the standard action video to the user terminal corresponding to the target user to prompt the target user to adjust the action according to the standard action video, etc., so as to further improve the accuracy of the production operation.

[0220] The target user can view the detection prompt information in the user interface of the user terminal 702 to obtain the detection summary results or adjust his own operations according to the action adjustment prompt information, and view the standard action video for learning.

[0221] like Figure 8 , which is a schematic diagram of a structure of an embodiment of a computing device provided by the present application, the device may include a storage component 801 and a processing component 802;

[0222] The storage component 801 can be used to store one or more computer program instructions, wherein the one or more computer program instructions are called and executed by the processing component 802 to implement Figure 1 The production operation test method shown, or Figure 2 The motion reference data generation method shown, or Figure 3 The information display method shown.

[0223] Of course, the above computing device may also include other components, such as input / output interfaces, communication components, etc.

[0224] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0225] It should be noted that the above computing device implements Figure 1 The production operation test method shown, or Figure 2In the case of the motion benchmark data generation method shown, it can be a physical device or an elastic computing host provided by a cloud computing platform, etc. It can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0226] The computing device described above implements Figure 3 In the case of the information display method shown, it can be implemented as an electronic device. The electronic device can refer to a device used by a user and having the functions of surfing the Internet, computing, communicating, etc. required by the user, such as a mobile phone, a tablet computer, a personal computer, a wearable device, etc. It is understandable that the above electronic device may also include a display component, an input / output interface, a communication component, and other components, which will not be repeated.

[0227] In one or more of the above embodiments, the processing component may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component can also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0228] The storage component is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0229] The display component may be an electroluminescent (EL) element, a liquid crystal display or a micro display having a similar structure, or a retinal direct display or a similar laser scanning display.

[0230] The present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a computer, Figure 1 The production operation test method shown, or Figure 2 The motion reference data generation method shown, or Figure 3 The information display method shown. The computer readable medium may be included in the computing device described in the above embodiment; or it may exist independently without being assembled into the computing device.

[0231] The computer-readable storage medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above.

[0232] The present application also provides a computer program product, which includes a computer program carried on a computer-readable storage medium, and when the computer program is executed by a computer, it can achieve Figure 1 The production operation test method shown, or Figure 2 The motion reference data generation method shown, or Figure 3 The information display method shown.

[0233] In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, various functions defined in the system of the present application are performed.

[0234] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0235] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0236] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0237] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A production operation detection method, characterized in that: include: Obtain production operation videos captured by video capture equipment; The production operation video is obtained by capturing the production actions performed by the target user by the video capture device; Segmenting the production operation video to obtain a video segment corresponding to a target production process; identifying production action data in the video clip; The production action data is compared with the action reference data corresponding to the target production process to obtain a detection result.

2. The method according to claim 1, characterized in that The production operation video is obtained by the video acquisition device capturing the production action performed by the target user based on the production device; the method further includes: Acquiring equipment operation data of the production equipment; The step of segmenting the production operation video to obtain a video clip corresponding to the target production process includes: Performing time alignment processing on the equipment operation data and the production operation video; According to the equipment operation data corresponding to the target production process, the production start time and the production end time corresponding to the target production process are determined, and according to the production start time and the production end time, the video segment corresponding to the target production process is obtained from the production operation video segmentation.

3. The method according to claim 1, characterized in that The step of segmenting the production operation video to obtain a video clip corresponding to the target production process includes: Identify the starting production action and the ending production action corresponding to the target production process from the production operation video; According to the starting production action and the ending production action, a video segment corresponding to the target production process is obtained from the production operation video segmentation.

4. The method according to claim 1, characterized in that: The identifying the production action data in the video clip comprises: Using an action recognition model to identify an action sequence in the video clip and a time interval between adjacent production actions in the action sequence; The action recognition model is obtained based on production sample videos and training labels.

5. The method according to claim 4, characterized in that The action reference data includes action sequence reference data and interval reference data between adjacent production actions; The step of comparing the production action data with the action reference data corresponding to the target production process to obtain the detection result comprises: Comparing the action sequence with corresponding action sequence benchmark data to obtain sequence comparison data; Compare the time interval between two adjacent production actions with the corresponding interval reference data to obtain time comparison data; A detection result is obtained based on the sequence comparison data and the time comparison data.

6. The method according to claim 5, characterized in that The action reference data also includes production action error data and predetermined type of action; The step of comparing the production action data with the action reference data corresponding to the target production process to obtain the detection result further comprises: Comparing the production action in the action sequence with the production action error data to obtain action comparison data; Identify whether the action sequence includes the predetermined type of action, and obtain an action recognition result; The obtaining of the detection result based on the sequence comparison data and the time comparison data comprises: A detection result is obtained based on the sequence comparison data, the time comparison data, the action comparison data and the action recognition result.

7. The method according to claim 4, characterized in that Also includes: Obtaining production sample videos and training labels; the training labels include sample motion data formed by the interaction between the limbs and the production materials and / or production equipment; The action recognition model is trained using the production sample video and the training labels.

8. The method according to claim 6, characterized in that Also includes: Obtaining a standard action video corresponding to the target production process; Using the action recognition model to identify the action standard sequence in the standard action video and the time standard interval between adjacent production actions in the action standard sequence; Generate the action sequence benchmark data based on the action standard sequence and the execution order error data and / or execution number error data of the production action; Generate the interval reference data based on the time standard interval and interval error data between adjacent production actions in the action standard sequence; generating the motion error data based on motion element errors and / or motion position errors of the production motion in the motion standard sequence; The predetermined type of action in the standard sequence of actions is determined.

9. The method according to claim 1, characterized in that: Also includes: Generate detection prompt information based on multiple detection results corresponding to multiple video clips corresponding to the target production process within a preset time range; The target user is notified of the detection prompt information.

10. The method according to claim 6, characterized in that Also includes: If it is determined based on the detection result that there is a risk action in the action sequence, generating risk warning information; Notify the target user of the risk warning information.

11. The method according to claim 1, characterized in that: Also includes: Obtaining a binding request sent by a user terminal; the binding request includes a first device identifier corresponding to the video acquisition device; Establishing an association relationship between the first device identifier and a user identifier of a target user corresponding to the user terminal; After obtaining the production operation video acquired by the video acquisition device, the method further includes: According to the first device identifier of the video acquisition device, searching the association relationship to determine the corresponding user identifier; The method further comprises: A corresponding relationship between the user identification and the detection result is established.

12. The method according to claim 11, characterized in that Also includes: Based on the user identifier, detection prompt information corresponding to the detection result is sent to the corresponding user terminal.

13. The method according to claim 8, characterized in that Also includes: The standard action video is sent to a user terminal corresponding to the target user.

14. The method according to claim 1, characterized in that The step of segmenting the production operation video to obtain a video clip corresponding to the target production process includes: Segment the production operation video to obtain video clips corresponding to a plurality of production processes; Any production process is taken as a target production process to obtain a video clip corresponding to the target production process.

15. An information display method, characterized in that: include: Based on the first device identifier of the video acquisition device, sending a binding request to the server; The binding request is used to request the server to establish an association relationship between the first device identifier and the user identifier of the target user, the association relationship is used to find the user identifier corresponding to the first device identifier of the video acquisition device, and the server sends the detection prompt information based on the user identifier; The detection prompt information is generated based on the detection result corresponding to the target production process; the detection result is obtained by comparing the production action data identified from the video clip corresponding to the target production process with the action reference data, and the video clip is obtained by the server end segmenting the production operation video collected by the video acquisition device; Obtaining the detection prompt information sent by the server; The detection prompt information is displayed in the user interface.

16. The method according to claim 15, characterized in that Also includes: Obtain the standard action video corresponding to the target production process sent by the server, and play the standard action video.

17. The method according to claim 15, characterized in that Also includes: In response to a scanning operation on a medium carrying the first device identification in the video acquisition device, a first device identification of the video acquisition device is acquired.

18. A computing device, characterized in that It comprises a storage component and a processing component; the storage component stores one or more computer program instructions, the computer program instructions are called and executed by the processing component, and the processing component executes the one or more computer program instructions to implement the production operation detection method as described in any one of claims 1 to 14, or the information display method as described in any one of claims 15 to 17.

19. A computer-readable storage medium, characterized in that: A computer program is stored, and the computer program is executed by a computer to implement the production operation detection method according to any one of claims 1 to 14, or the information display method according to any one of claims 15 to 17.

20. A computer program product, characterized in that A computer program is stored, and when the computer program is executed by a computer, the production operation detection method according to any one of claims 1 to 14 or the information display method according to any one of claims 15 to 17 is implemented.

Images