Detection method, device and system

Abstract

The invention discloses a detection method, device and system, and relates to the technical field of neural networks, and the method comprises the steps: collecting a first image of a to-be-detected object when the to-be-detected object is located at a target position corresponding to a target station; obtaining a target detection model matched with the target station and the to-be-detected object; and detecting the first image through the target detection model, and determining that a to-be-detected object passes the detection under the condition that the to-be-detected object is detected to meet a target condition. According to the invention, the method can be suitable for different stations for detection, and does not use other external equipment for detection, thereby reducing the detection cost, and increasing the application scenes of detection at the same time.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and more specifically, to a detection method, device and system. Background Art

[0002] Currently, the quality of products on the production line can be tested by testing equipment, but the testing equipment cannot adapt to the testing of different tasks at different workstations. Therefore, how to test different tasks corresponding to different workstations on the production line has become an urgent problem to be solved. Summary of the invention

[0003] The present application proposes a detection method, device and system.

[0004] In a first aspect, an embodiment of the present application provides a detection method, comprising: when an object to be detected is in a target position corresponding to a target workstation, acquiring a first image including the object to be detected; acquiring a target detection model that matches the target workstation and the object to be detected; detecting the first image through the target detection model, and determining that the object to be detected has passed the detection when it is detected that the object to be detected meets the target conditions.

[0005] In a second aspect, an embodiment of the present application further provides a detection device, comprising: a collection unit, configured to collect a first image including the object to be detected when the object to be detected is at a target position corresponding to a target station;

[0006] An acquisition unit is used to acquire a target detection model that matches the target workstation and the object to be detected; a detection unit is used to detect the first image through the target detection model, and when it is detected that the object to be detected meets the target condition, determine that the object to be detected has passed the detection.

[0007] In the third aspect, an embodiment of the present application also provides a detection system, including: a processor and an image acquisition unit; the processor is connected to the image acquisition unit; the image acquisition unit is used to collect a first image including an object to be detected; the processor is used to detect the first image based on the method described in the first aspect.

[0008] The detection method, device and system provided in the embodiments of the present application first collect a first image including the object to be detected, and determine a target detection model that matches the target workstation and the object to be detected, so that the first image including the object to be detected can be detected by the target detection model to determine whether the object to be detected meets the target conditions, and when it is determined that the target conditions are met, it is determined that the object to be detected has passed the detection, so that it can be applied to different workstations for detection and use other external equipment for detection as little as possible, thereby reducing the detection cost and increasing the applicable scenarios of the detection. Flexible deployment is more conducive to promotion.

[0009] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 An application scenario diagram of the detection method provided in an embodiment of the present application is shown;

[0012] Figure 2 A method flow chart of the detection method provided in an embodiment of the present application is shown;

[0013] Figure 3 A method flow chart of a detection method provided by another embodiment of the present application is shown;

[0014] Figure 4 A method flow chart of a detection method provided by another embodiment of the present application is shown;

[0015] Figure 5 A schematic diagram showing the similarity between the unit to be detected and the standard content provided by an embodiment of the present application is shown;

[0016] Figure 6 A schematic diagram of marking a detection area of ​​a detection unit to be detected provided in an embodiment of the present application is shown;

[0017] Figure 7 A method flow chart of a detection method provided by another embodiment of the present application is shown;

[0018] Figure 8 A schematic diagram showing whether the first image provided by an embodiment of the present application matches the standard document;

[0019] Fig. 9 A method flow chart of a detection method provided by yet another embodiment of the present application is shown;

[0020] Fig.10 The structure block diagram of the detection device provided by the embodiment of the present application is shown;

[0021] Fig.11 The structure block diagram of the detection system provided by the embodiment of the present application is shown;

[0022] Fig.12 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0024] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0025] Deep Learning: Deep Learning is a branch of machine learning. It refers to a method of machine learning using multi-layer neural networks. It learns the inherent rules and representation levels of sample data. The ultimate goal is to enable machines to have analytical learning capabilities like humans and to recognize data such as text, images, and sounds.

[0026] Convolutional Neural Networks: Convolutional Neural Networks (CNN) is a deep learning model that has achieved great success in many fields such as image and video analysis, image classification, object detection, facial recognition, etc. CNN processes and analyzes visual data by simulating the working principle of the human visual system.

[0027] Large Language Model: Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data that can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue.

[0028] At present, with the development of computer technology, automated testing equipment can be used to test products or employee operations at each workstation on a product production line.

[0029] However, the inventors found in their research that the use of automated testing equipment cannot meet the testing requirements of different workstations on the production line, that is, it is not suitable for testing at different workstations, resulting in the inability to perform testing at different workstations, and due to the high cost of automated testing, the cost of testing is high.

[0030] Therefore, in order to solve or partially solve the above problems, the present application provides a detection method, device and system.

[0031] See also Figure 1 , Figure 1 An application scenario diagram of the detection method provided by an embodiment of the present application is shown, and the application scenario diagram includes a detection result control interaction module 110, a detection data recording and alarm module 120, an image acquisition and analysis processing module 130, a model management and control module 140 and a display module 150, wherein the detection result control interaction module 110 is used to communicate and interact with different devices or software; the detection data recording and alarm module 120 is used to record data on images of detection anomalies and to alarm when anomalies are detected, and can record abnormal images and original images in the detection station according to the abnormal time, wherein the detected abnormal images are used to analyze abnormal phenomena, and the original images are used to optimize the detection model; the image acquisition and analysis processing module 130 is used to acquire and analyze images in real time; the model management and control module 140 is used to determine the target detection model that matches the target workstation and the object to be detected; the display module 150 is used to display the detection results.

[0032] Optionally, in the test result control interaction module 110, communication can be performed through COM and TCP. The program can distinguish COM and TCP communication control in the form of configuration files. The COM method is mainly used to control the detection of workstations without equipment (for example, the mobile phone products on the belt line flow backwards through the belt, and the detection can be directly installed on the belt to control the start and stop of the belt. After the product on the belt reaches the detection position, it receives the COM result instruction of the detection software. If the detection software result is not passed, the belt stops and the buzzer alarms). The TCP method is mainly used to control the communication with the test software on the current device side. It is mainly used to regard the test result as a test item of this workstation. If the result of this test item is not passed, the final result of this device is not passed. Only those with a final result of passing will be allowed to pass the station.

[0033] Optionally, in the interactive format of communicating in TCP mode, multiple USB camera detections are supported through local TCP communication, and different USB cameras are distinguished through TCP ports (if there are multiple cameras on the device side, such as USB cameras with serial numbers 0, 1, 2, and 3, to prevent the camera sequence from being garbled after the device computer is powered off and restarted, the USB camera is distinguished by writing the corresponding hardware ID, so that the serial number and the hardware ID are mapped for identification, and the serial number currently on this processing device is obtained according to the required serial number. For example, 4 detection positions are added to a device, the upper left and right, and the lower left and right, corresponding to serial numbers 0, 1 and 3, 4 respectively. The corresponding hardware ID information is checked according to the serial number required in the configuration file at this time, and then the serial number corresponding to the USB camera of the hardware ID signal in the processing device is checked, so as to determine the actual serial number corresponding to the current processing device, and avoid confusion in detecting cameras). The device software only needs to send the TCP of the corresponding port to obtain the detection result when the product moves to the added USB camera detection position, and the detection software is real-time detection, and will immediately return the detection result corresponding to the target station.

[0034] See also Figure 2 , Figure 2 A flow chart of a detection method provided in an embodiment of the present application is shown. The detection method can be applied Fig.11 The detection system may specifically be a processor of the detection system as an execution subject of the detection method. The detection method may include steps S110 to S130.

[0035] Step S110: when the object to be detected is at a target position corresponding to a target workstation, a first image including the object to be detected is acquired.

[0036] As a method, on the product production line, the staff at the workstation may make mistakes during the operation of the product, resulting in the product quality not meeting the product standards. In order to ensure the production quality of the product, the neural network can be used to identify the object to be inspected at the target workstation to determine whether the object to be inspected can pass the inspection. When it is determined that the object to be inspected has passed the inspection, it is determined that the quality of the product in the production process meets the product standards, ensuring that there are no abnormalities in the information and operation of each workstation on the production line, thereby improving the product's Quality Error Proof (QEP) level and preventing quality problems in the production process.

[0037] Optionally, a sensing device may be provided on the target station, so that the sensing device can be used to determine whether the object to be detected is at the target position of the target station. Optionally, an image acquisition device may be provided, which can monitor the target position of the target station, so that the image acquisition device can monitor in real time whether there is an object to be detected at the target position of the target station.

[0038] Optionally, when the image acquisition device monitors in real time that there is an object to be detected at the target position of the target station, the image acquisition device acquires a first image at the target position of the target station, and the first image includes the object to be detected. Optionally, the object to be detected may be a staff member, a product, or a product operation guide, etc., wherein the product operation guide may be a guide for guiding staff members to operate the product, so that the staff members can correctly operate the product so that the product can meet the factory standards.

[0039] Optionally, when it is determined that the target position of the target station is blocked or there is an object, the first image including the object to be detected in the target position of the target station is collected. Optionally, the target station can be a station set by the user to be required to perform quality inspection or operator operation inspection, and the target position can be the position where the product is prevented when the staff operates at the target station, or the position of the operating table where the staff operates the station, or the position where the operation guide is posted on the station.

[0040] Step S120: Acquire a target detection model that matches the target workstation and the object to be detected.

[0041] As a method, since different detection models correspond to different objects to be detected, and the detection objects to be detected at different workstations are different, it is necessary to first determine the target detection model that matches the target workstation and the object to be detected based on the target workstation and the object to be detected in the first image.

[0042] Optionally, a model set can be set in advance, which includes multiple models for performing different detection tasks, and corresponding task identifiers are set for the multiple models, and corresponding task identifiers are set for different workstations and different objects. In this way, a matching target detection model can be determined in the model set by determining the representation corresponding to the target workstation and the task identifier corresponding to the object to be detected at the target position of the target table workstation.

[0043] Optionally, a model having the same task identifier as the task identifier of the target station and the task identifier corresponding to the task to be detected can be determined as the target detection model. Optionally, the task identifier of the object to be detected in the first image can be determined by identifying the first image, and the task identifier of the target station can be determined based on information such as the location of the image acquisition device of the first image, wherein the user pre-sets the corresponding task identifiers of the target stations corresponding to the image acquisition devices at different positions. For example, for stations 1, 2, and 3 on the same production line, the target task of station 1 is that the staff operates 1 on the product, the target task of station 2 is that the staff operates 2 on the product, and the target task of station 3 is to perform quality inspection on the product. Therefore, the target task of station 1 is to detect whether operation 1 is wrong, the target task of station 2 is to detect whether operator 2 is wrong, and the target task of station 3 is to detect the quality of the product. The corresponding object to be detected at station 1 is the operation 1 of the station staff, the object to be detected at station 2 is the operation 2 of the staff, and the object to be detected at station 3 is the product.

[0044] Step S130: Detecting the first image using the target detection model, and determining that the object to be detected passes the detection when it is detected that the object to be detected meets the target condition.

[0045] As a method, detecting the first image through a target detection model can be to determine whether the first image is an abnormal graphic, wherein the abnormal graphic can be whether the object to be detected in the first image meets the target condition. After the target detection model outputs the detection result, it is determined that the object to be detected has passed the detection when the detection result indicates that the object to be detected meets the target condition.

[0046] Optionally, the target condition may be whether the product appearance meets factory requirements or whether the operator's operation sequence on the product is correct. The target condition is related to the specific object to be detected. Therefore, the detection task of the target detection model can be determined first, and the object to be detected can be determined based on the detection task, so as to determine the target condition corresponding to the object to be detected.

[0047] Optionally, the detection result output by the target detection model after detecting the first image can be an image annotated after the object to be detected in the first image is detected. If there is an object with the target suffix in the image annotation in the output detection result, it can be determined that the object to be detected does not meet the target condition; if the object with the target suffix is ​​not detected, it can be determined that the object to be detected meets the target condition. Optionally, when it is determined that the object to be detected does not meet the target condition, it can be prohibited to transfer the product to the next workstation or the next processor, so as to avoid product quality problems caused by defective products passing through the station.

[0048] In an embodiment of the present application, when the object to be detected is in the target position corresponding to the target workstation, a first image including the object to be detected is first acquired, and a target detection model matching the target workstation and the object to be detected is determined, so that the first image including the object to be detected can be detected by the target detection model to determine whether the object to be detected meets the target conditions, and when it is determined that the target conditions are met, it is determined that the object to be detected has passed the detection, so that it can be applied to different workstations for detection, and other external equipment can be used as little as possible for detection, thereby reducing the detection cost and increasing the applicable scenarios of the detection. Flexible deployment is more conducive to promotion.

[0049] See also Figure 3 , Figure 3 A method flow chart of a detection method provided in an embodiment of the present application is shown. The detection method can be applied to Fig.11 The detection system shown can specifically use the processor of the detection system as the execution subject of the detection method. The detection method can include steps S210 to S260, wherein the first image is multiple, and the object to be detected includes a workstation employee.

[0050] Step S210: Determine the standard operating procedure corresponding to the target workstation.

[0051] As a method, when the object to be tested is a workstation employee, the operation of the workstation employee is tested, and each step of the workstation employee's operation can be tested, and the standard operating procedure corresponding to the target workstation can be obtained. Optionally, the standard operating procedure can be an operating procedure set for the workstation employee at each workstation. If the workstation employee operates the product according to the standard operating procedure at the corresponding workstation, the quality of the product can be guaranteed to a certain extent.

[0052] Optionally, the target workstation and its corresponding standard operating procedure may be associated and stored in advance, so that the standard operating procedure corresponding to the target workstation may be obtained in the database based on the workstation identifier of the target workstation.

[0053] Step S220: Obtain the standard duration corresponding to the standard operating procedure.

[0054] As a method, the corresponding operation time can be set for each process in the standard operation process in advance, so as to obtain the standard time corresponding to the standard operation process based on the operation time corresponding to each process. Among them, the standard time is used to indicate the time required for employees to operate the product according to the standard operation process. The standard time can be specifically set based on different standard operation processes, and is not specifically limited here.

[0055] Step S230: Determine the first moment when the workstation employee is at the target position corresponding to the target workstation.

[0056] As a method, in order to determine whether an error occurs during the entire process of the workstation employee operating the product, it is necessary to collect images of the entire process of the workstation employee operating the product, so that it can be determined whether the workstation employee's operation is the same as the standard operating procedure by detecting the images of the entire process of the workstation employee operating the product. Therefore, to ensure the accuracy of the images of the entire process of the workstation employee operating the product, the first moment when the workstation employee is at the target position corresponding to the target workstation can be determined first, so that the image of the entire process of the workstation employee operating the product can be obtained based on the first moment. Optionally, the first moment can be determined by detecting when the workstation employee is at the target position of the target workstation through an image acquisition device.

[0057] Step S240: From the first moment to the standard time, continuously collect a plurality of second images including the employee at the workstation, each of the second images corresponds to one of the first images.

[0058] As a method, in order to monitor the entire operation process of the workstation employee in real time, images of the entire process from the workstation employee starting the workstation at the target position of the target workstation to the workstation employee ending the operation at the target position can be continuously collected.

[0059] Optionally, in order to avoid the existence of multiple duplicate images in multiple second images, the acquisition cycle can be set in advance, and multiple second images of employees at the workstation within a standard time period from the first moment to the end are acquired according to the acquisition cycle, thereby avoiding the problems of low detection efficiency and large memory usage caused by detecting every second image that is the first image.

[0060] Optionally, an image acquisition device (such as a camera) may be used to capture video of the workstation employee from the first moment to the standard duration, and then multiple second images may be determined in the captured video, so that each second image is used as the first image.

[0061] Step S250: Acquire a target detection model that matches the target workstation and the object to be detected.

[0062] Step S260: Detect the first image using the target detection model, and when it is detected that the object to be detected meets the target condition, determine that the object to be detected has passed the detection.

[0063] The specific steps of step S250 to step S260 may refer to step S120 to step S130, which will not be described in detail here.

[0064] In some embodiments, the standard operating procedure corresponds to multiple process nodes, and step 260 includes:

[0065] Step S261: Determine at least one key process node from the plurality of process nodes.

[0066] As a way to accurately determine whether the station employees can pass the test, at least one key process node can be determined from the multiple process nodes in the standard operation process corresponding to the target station. The key process node is the key step for the station employees to operate the product in the standard operation process. If the station employees make mistakes in the key step, it will greatly cause the product to not meet the factory requirements, resulting in major quality issues for the product. For example, if a screw falls off in the battery compartment of a mobile phone, a major quality incident of the battery being crushed and exploded will occur. Therefore, the material assembly of the battery compartment can be used as a key process node in the process of operating the battery of the mobile phone.

[0067] Step S262: Detect each of the first images based on the first standard image corresponding to each of the key process nodes through the target detection model, and determine that the workstation employee has passed the detection when it is detected that the workstation employee meets the target condition.

[0068] As a way, in order to accurately determine whether the employees at the workstation are operating the product correctly during the process of operating the product, standard images corresponding to the operations performed by the employees at each key process node can be collected in advance, so that the target detection model can be used to perform detection based on the standard images corresponding to each key process node and the images collected from the employees at the target workstation in real time, so as to determine whether the operations of the employees at the workstation meet the target conditions.

[0069] Optionally, when at least one key process node is determined, whether the first standard image of each key process node matches the first image can be determined through a target detection model that matches the workstation employee and the target workstation, or the similarity between the first standard image and the first image can be determined through the target detection model to determine whether the workstation employee meets the target conditions.

[0070] Optionally, the target condition corresponding to the workstation employee may be a condition corresponding to a key process node, so as to determine whether the workstation employee meets the corresponding target condition at different key process nodes. Optionally, the target condition may be used as a screening condition in the target detection model, so as to determine whether the selected condition exists in the target condition during the detection of the first image based on the first standard image, so as to determine whether the workstation employee meets the target condition.

[0071] In some embodiments, step S262 includes:

[0072] Step S2621: Acquire a first image that matches a first standard image corresponding to any one of the key process nodes as the third image.

[0073] As a method, since the operations performed by the staff at the same workstation on the product may be multiple operations, and each operation has its corresponding key process node, in order to determine whether the target conditions are met in the process of the staff operating the product, it is necessary to confirm whether each key process node meets its corresponding target conditions, and whether the corresponding operation sequence is correct also affects the determination of whether the target conditions are met. Therefore, in order to determine whether the operation of the staff at the workstation is correct during the entire process, the first image that matches the first standard image corresponding to any key process node can be obtained from the multiple first images as the third image.

[0074] Optionally, any first image whose similarity with each key process node among the multiple images is greater than a similarity threshold may be determined as the third image.

[0075] Step S2622: respectively obtain the first number of the third images and the second number of the key process nodes.

[0076] As a method, since each key process node in the entire operation process of the workstation employee needs to meet the corresponding target conditions in order to determine whether the workstation employee's operation of the product meets the target conditions, therefore, by determining the first number of any third images matching each key process node and the second number of key process nodes, it can be determined whether the first number and the second number are the same to determine whether the entire operation process of the workstation employee meets the operation standards.

[0077] Step S2623: When the first number is equal to the second number, the first order of the third images is determined based on the order of the timestamps of the third images, and the second order of the third images is determined based on the order of occurrence of the key process nodes corresponding to each of the third images.

[0078] As a way, when it is determined that the first quantity is equal to the second quantity, it can be determined that the entire process of the employee's operation on the product at the workstation complies with standard operations. However, the quality of the product is also related to the operation sequence of each key process node by the employee at the workstation. If the operation sequence is wrong, it will still cause quality problems in the product. Therefore, it is also necessary to determine whether the operation sequence of the employee at each key process node complies with standard operations.

[0079] Optionally, since the third image is one of the first images that matches the first standard image, the order in which employees operate on the product can be determined by the timestamp corresponding to each third image, and then the first order can be obtained, that is, the first order indicates the order in which employees at the workstation operate on the product, and in the standard operating procedure, the order of each key process node will be set, that is, the second order. Therefore, the second order refers to the order of standard operations on the product.

[0080] Step S2624: When the first sequence is the same as the second sequence, determine that the workstation employee has passed the inspection.

[0081] As a way, when it is determined that the first sequence is the same as the second sequence, it can be determined that the station employee operates the product in full accordance with the standard operating procedures, and also operates the product in accordance with the specific operations of the standard operating procedures. Therefore, it can be determined that the station employee's operation of the product meets the corresponding target conditions, that is, the station employee operates without abnormality in this cycle. Therefore, it can be determined that the station employee has passed the test, thereby ensuring the quality of the product. For example, at the station where the station employee installs a camera on the product, the station employee takes the camera as a key process node, the installation is a key process node, and the installation is a key process node. By detecting the first sequence of the third image corresponding to these three key process nodes and the second sequence corresponding to the occurrence of these three key process nodes, whether they are all carried out in the order of 1, 2, and 3, it is determined that they are all carried out in the order of 1, 2, and 3, and then it is determined that the station employee has passed the test.

[0082] Optionally, if the first sequence is different from the second sequence, it is determined that the workstation employee has failed the inspection. At this time, a corresponding alarm message can be generated, and an alarm prompt can be given based on the alarm message to avoid waste of resources caused by subsequent processing of products with quality problems.

[0083] In this embodiment, when the object to be detected is a workstation employee, the standard operating procedure corresponding to the target workstation is determined, so as to determine the standard duration corresponding to the standard operating procedure. Then, based on the standard duration and the first moment when the employee is in the target position corresponding to the target workstation, multiple second images including the workstation employee can be acquired. Then, the workstation employee can be detected based on the multiple second images, thereby realizing the detection of the workstation employee's behavior and ensuring the accuracy of the detection.

[0084] See also Figure 4 , Figure 4 A method flow chart of a detection method provided in an embodiment of the present application is shown. The detection method can be applied to Fig.11 The detection system shown in the figure can specifically use the processor of the detection system as the execution subject of the detection method. The detection method can include steps S310 to S350, wherein the object to be detected includes a unit to be detected in the product to be detected.

[0085] Step S310: Acquire a second standard image corresponding to the target workstation, wherein the second standard image includes standard content.

[0086] As a way, in order to ensure the quality of the product, the unit to be inspected in the product to be inspected can be inspected, wherein the unit to be inspected can be a module or component in the product to be inspected. Each component in the unit to be inspected has its corresponding position. In order to ensure the quality of the product, a corresponding standard image can be set for each unit to be inspected. The standard content in the standard image can be used to guide the station employees to perform installation and other operations. Therefore, the second standard image corresponding to the target station can be obtained based on the components that need to be assembled or operated corresponding to the target station. Optionally, the standard content in the second standard image can be the position of each component or module in the unit to be inspected after the station employee of the target station accurately installs or operates it.

[0087] Step S320: Detecting, by means of the target detection model, whether the unit to be detected in the product to be detected in the first image matches the standard content.

[0088] As a method, in order to determine whether the unit to be inspected in the product to be inspected meets the target condition, the target detection model can be used to detect whether the unit to be inspected in the product to be inspected in the first image matches the standard content in the second standard image.

[0089] Optionally, the target detection model can be used to extract features from the same positions in the first image and the second standard image to obtain corresponding features, and then determine the similarity between the features extracted in the first image and the second standard image to determine whether the unit to be inspected in the product to be inspected in the first image matches the standard content.

[0090] Optionally, the target detection model may first add a corresponding detection frame or detection suffix to the detection unit of the product to be detected in the first image, so as to distinguish the detection area through the detection frame or detection suffix, thereby improving the detection efficiency of the target detection model.

[0091] Optionally, after obtaining the first image, all units to be detected in the first image may be classified first, so that the first image can be detected quickly. Optionally, the units to be detected in the first image may be annotated to classify them, and different suffixes may be used to annotate different units to be detected to achieve classification, and then when the target detection model detects the first image, the units to be detected are determined by detecting the suffix name, and whether the units to be detected match the standard content is determined.

[0092] Optionally, when the target detection model detects whether the unit to be detected in the product to be detected in the first image matches the standard content, it may determine whether the unit to be detected matches the standard content by the similarity between the same position or the same unit. Figure 5 As shown, in Figure 5 In the process, the installation position of the component in the device of the first image is detected, and the target detection model is used for detection, and a detection image is output. The detection image includes multiple units to be detected. By determining the similarity between the units to be detected and the standard content, that is, the numerical part in the figure, it is determined whether the units to be detected in the product to be detected match the standard content. Different suffixes (such as pm, sp, cm1, cm2, rf, spk, etc. in the figure) indicate different types of units to be detected.

[0093] Step S330: When the unit to be detected matches the standard content, it is determined that the unit to be detected in the product to be detected passes the detection.

[0094] As a way, it can be determined that the unit to be detected matches the standard content by determining whether the similarity between the unit to be detected and the standard content is greater than a similarity threshold, so as to determine that the unit to be detected in the product to be detected passes the test. Optionally, when it is determined that the similarity between the unit to be detected and the standard content is greater than or equal to the similarity threshold, it is determined that the unit to be detected matches the standard content; when it is determined that the similarity between the unit to be detected and the standard content is less than the similarity threshold, it is determined that the unit to be detected does not match the standard content.

[0095] Optionally, in order for monitoring personnel to clearly view or the system to clearly display that the unit to be inspected in the product to be inspected has passed the inspection, the word "PASS" can be displayed in the picture, and there is no need to add other image annotations in the first image, or for easy viewing, the type of the unit to be inspected in the product to be inspected in the first image can be marked, and the inspection area of ​​the unit to be inspected can be marked, and the inspection area can be distinguished by adding the type name with the suffix "area" as a label, such as Figure 6 As shown, phone_area is added to distinguish the detection area, so as to avoid misdetection caused by detecting other parts in the first image.

[0096] Optionally, when it is determined that the unit to be inspected does not match the standard content, it is determined that the unit to be inspected in the product to be inspected has failed the inspection. In order to be able to determine the specific non-matching unit to be inspected, it can be distinguished by adding an abnormal suffix name (for example, ng) in the first image. This makes it convenient for the workstation employees to check and process the unit to be inspected based on the abnormal suffix name, and can directly determine whether the unit to be inspected of the product to be inspected has passed the inspection based on whether there is an abnormal suffix name.

[0097] Step S340: Acquire a target detection model that matches the target workstation and the object to be detected.

[0098] Step S350: Detect the first image using the target detection model, and when it is detected that the object to be detected meets the target condition, determine that the object to be detected has passed the detection.

[0099] The specific steps of step S340 to step S350 may refer to step S120 to step S130, which will not be described in detail here.

[0100] In this embodiment, when the object to be inspected includes a unit to be inspected in the product to be inspected, the target detection model is used to detect whether the unit to be inspected in the product to be inspected in the first image matches the standard content included in the second standard image corresponding to the target workstation, so as to determine whether the unit to be inspected passes the inspection, thereby realizing product inspection and ensuring product quality.

[0101] See also Figure 7 , Figure 7 A method flow chart of a detection method provided in an embodiment of the present application is shown. The detection method can be applied to Fig.11 The detection system shown may specifically use a processor of the detection system as an execution subject for executing the detection method. The detection method may include steps S110 to S450, wherein the object to be detected includes an operation instruction document.

[0102] Step S410: When the object to be detected is at a target position corresponding to a target workstation, a first image including the object to be detected is acquired.

[0103] Step S420: Acquire a target detection model that matches the target workstation and the object to be detected.

[0104] The specific steps of step S410 to step S420 may refer to step S110 to step S120, which will not be described in detail here.

[0105] Step S430: Determine the standard text corresponding to the target workstation.

[0106] As a way to ensure product quality, it is necessary to detect text information such as operating instructions or product information on each workstation to avoid product quality problems caused by incorrect operating instructions or incorrect product information. The standard text corresponding to the target workstation can be determined first, so that the operating instruction document can be detected based on the standard text. Optionally, the standard text can be a text pre-stored in a database.

[0107] Step S440: Perform text recognition on the operation instruction document in the first image through the target detection model to obtain the text to be detected corresponding to the operation instruction document.

[0108] As a method, since the first image is an image captured in real time, the target detection model can be used to perform document recognition on the target position in the first image, so as to determine the text to be detected corresponding to the operation instruction document, wherein the target position is the position corresponding to the operation instruction document at the target workstation.

[0109] Step S450: when it is detected that the text to be detected matches the standard text, it is determined that the operation instruction document passes the detection.

[0110] As a method, the context similarity between the text to be detected and the standard text can be determined to determine whether the text to be detected matches the standard text. If the context similarity between the text to be detected and the standard text is greater than or equal to a preset value, it is determined that the text to be detected matches the standard text; if the context similarity between the text to be detected and the standard text is less than a preset value, it is determined that the text to be detected does not match the standard text.

[0111] Optionally, the inspection frequency can be set for the operating instruction document. It can be set according to actual needs. For example, when the operating instruction document is an operating guide, it can be tested once when the detection function is turned on; when the operating instruction document is the operating guide information of the product (for example, key information such as nameplate stickers on mobile phones and ten-piece stickers on packaging), it can be set to be tested every time for each product.

[0112] Optionally, all text information can be decomposed into multiple text test items, and the target detection model can be asked questions to obtain a conclusion of yes or no. Then, based on the conclusion of yes or no, the target detection model makes multiple judgments and displays PASS or FAIL and detailed information of the target detection model for the operation guide document. Only when all items are PASS can the operation guide document be determined to have passed the test. Optionally, the target detection model can be a large language model, and fine-tuning can be performed for information security and special content of the large model. The large model tool only needs to request through the API interface, and the large model can support multiple languages.

[0113] Optionally, in order to facilitate the detection of the operation instruction document, a separate large model information detection software can be encapsulated. After the large model information detection software obtains the first image and the standard document, it transmits the detection result through COM or TCP. Only after the detection result is passed, the product can pass the station, such as Figure 8 As shown, when the product fails the test (ie, FAIL), it is determined that the operation instruction document has passed the test and the product is not allowed to pass the station.

[0114] In this embodiment, when the object to be detected includes an operation instruction document, text recognition is performed on the operation instruction document in the first image through the target detection model to obtain the text to be detected corresponding to the operation instruction document, so as to determine whether the text to be detected matches the standard text corresponding to the target workstation, thereby determining whether the operation instruction document passes the detection and realizing detection in the text dimension, thereby ensuring the accuracy of detection in the text dimension.

[0115] See also Fig. 9 , Fig. 9 A method flow chart of a detection method provided in an embodiment of the present application is shown. The detection method can be applied to Fig.11 The detection system shown may specifically use a processor of the detection system as an execution subject for executing the detection method. The detection method may include steps S510 to S540.

[0116] Step S510: When the object to be detected is at a target position corresponding to a target workstation, a first image including the object to be detected is acquired.

[0117] Step S520: Obtain the target type of the target workstation.

[0118] As a method, different workstations perform different operations. When inspecting all workstations, it is necessary to determine the operations to be performed at the target workstation, so that the specific operations can be inspected through the corresponding detection model. In order to facilitate the determination of the target detection model, the target types of different workstations can be determined first. Optionally, the target type can be determined by identifying the first image or collecting the device information of the first image. Optionally, the target type can indicate a specific inspection task or inspection object. For example, the target type can be to inspect the appearance of the product or the behavior of the employees at the workstation.

[0119] In some embodiments, before step S530, the method further includes:

[0120] Step S610: obtaining training data corresponding to each category, wherein the training data includes a sample image and a label corresponding to the sample image, and the label includes a standard detection result for characterizing whether a sample object in the sample image passes the detection.

[0121] As a method, the sample images may include images of employees at different workstations operating the product and corresponding images of the product appearance. Optionally, the sample images include images that pass the inspection and images that fail the inspection.

[0122] Step S620: training the initial models corresponding to each category based on the sample images corresponding to the category, so as to obtain an initial detection result output by each initial model for representing whether the sample object passes the detection.

[0123] As a method, the initial model can be an image recognition model including a convolutional neural network and / or a fully connected neural network, and then the image recognition model of the corresponding category can be trained through the sample images corresponding to each category, so as to obtain the initial detection results of the sample images recognized by the image recognition models corresponding to different categories, wherein the initial detection results include the sample images passing the detection and the sample images failing the detection.

[0124] Step S630: Based on the difference between the initial detection result and the standard detection result, the initial model corresponding to the category is trained to reduce the difference.

[0125] As a method, when the initial detection result is different from the standard detection result, the error function can be determined based on the difference in detection scores or detection confidence between the area indicated as failing the detection in the initial detection result and the corresponding area in the standard detection result, and the parameters of the initial model can be adjusted based on the error value corresponding to the error function. The adjusted initial model is then trained again based on the sample image until the initial detection result is the same as the standard detection result, or the error function converges, thereby terminating the training of the initial model and obtaining this type of detection model.

[0126] Step S640: Use the trained initial model corresponding to each category as the detection model corresponding to the category.

[0127] As a method, since the initial model corresponding to each category after training can satisfy that the initial detection result is the same as the standard detection result or the difference between the initial detection result and the standard detection result is less than the difference threshold, it can be determined that the initial model corresponding to each category after training can be applied offline, and then the initial model corresponding to each category after training is used as the detection model corresponding to the category.

[0128] Step S650: construct a model set based on the detection model corresponding to each category.

[0129] As a way, in order to be able to detect different types of target stations on the product production line, a model set is constructed according to the detection model corresponding to each type, so that the corresponding target detection model can be determined and used in real time based on the station type corresponding to the target station, thereby improving detection efficiency.

[0130] Step S530: Acquire a detection model corresponding to the target type from a pre-built model set as the target detection model, wherein the model set includes a plurality of detection models, and each detection model corresponds to a type of workstation.

[0131] As a method, each detection model in the model set includes its own corresponding category identification. Therefore, after determining the target category, the detection model corresponding to the target category can be obtained in the pre-built model set based on the category identification of the target category to determine the target detection model.

[0132] Step S540: Detect the first image using the target detection model, and when it is detected that the object to be detected meets the target condition, determine that the object to be detected has passed the detection.

[0133] The specific steps of step S510 and step S540 may refer to step S110 and step S130, which will not be described in detail here.

[0134] In this embodiment, the target type of the target workstation is determined, thereby determining a target detection model based on the target type, thereby ensuring the accuracy of detection performed by the target detection model, and further ensuring the accuracy of the detection result.

[0135] See also Fig.10 , Fig.10 A structural block diagram of a detection device provided in an embodiment of the present application is shown. The detection device 400 includes: a collection unit 410, an acquisition unit 420 and a detection unit 430.

[0136] The acquisition unit 410 is used to acquire a first image including the object to be detected when the object to be detected is at a target position corresponding to the target station;

[0137] An acquisition unit 420 is used to acquire a target detection model that matches the target workstation and the object to be detected;

[0138] The detection unit 430 is used to detect the first image by using the target detection model, and determine that the object to be detected has passed the detection when it is detected that the object to be detected meets the target condition.

[0139] Optionally, there are multiple first images, and the objects to be detected include workstation employees. The detection unit 430 is also used to determine a standard operating procedure corresponding to the target workstation; obtain a standard duration corresponding to the standard operating procedure; determine the first moment when the workstation employee is at the target position corresponding to the target workstation; and continuously acquire multiple second images including the workstation employees from the first moment to the standard duration, each second image corresponding to one of the first images.

[0140] Optionally, the standard operating procedure corresponds to multiple process nodes, and the detection unit 430 is also used to determine at least one key process node from the multiple process nodes; each first image is detected based on the first standard image corresponding to each key process node through the target detection model, and when it is detected that the workstation employee meets the target conditions, it is determined that the workstation employee has passed the detection.

[0141] Optionally, the detection unit 430 is also used to obtain a first image that matches the first standard image corresponding to any one of the key process nodes as a third image; obtain a first number of the third images and a second number of the key process nodes respectively; when the first number is equal to the second number, determine a first order of the third images based on the order of the third images in timestamps, and determine a second order of the third images based on the order of occurrence of the key process nodes corresponding to each of the third images; when the first order is the same as the second order, determine that the workstation employee has passed the detection.

[0142] Optionally, the object to be inspected includes a unit to be inspected in the product to be inspected, and the detection unit 430 is also used to obtain a second standard image corresponding to the target workstation, wherein the second standard image includes standard content; detect whether the unit to be inspected in the product to be inspected in the first image matches the standard content through the target detection model; when the unit to be inspected matches the standard content, determine that the unit to be inspected in the product to be inspected has passed the inspection.

[0143] Optionally, the object to be detected includes an operation instruction document, and the detection unit 430 is further used to determine a standard text corresponding to the target workstation; text recognition is performed on the operation instruction document in the first image through the target detection model to obtain the text to be detected corresponding to the operation instruction document; when it is detected that the text to be detected matches the standard text, it is determined that the operation instruction document has passed the detection.

[0144] Optionally, the acquisition unit 410 is also used to obtain the target type of the target workstation; obtain a detection model corresponding to the target type from a pre-built model set as the target detection model, wherein the model set includes multiple detection models, and each detection model corresponds to a type of workstation.

[0145] Optionally, the detection device 400 also includes: a training data acquisition unit, used to acquire training data corresponding to each category, wherein the training data includes a sample image and a label corresponding to the sample image, and the label includes a standard detection result for characterizing whether a sample object in the sample image has passed the detection; a first training unit, used to train the initial model corresponding to each category based on the sample image corresponding to the category, so as to obtain an initial detection result output by each initial model for characterizing whether the sample object has passed the detection; a second training unit, used to train the initial model corresponding to the category based on the difference between the initial detection result and the standard detection result, so as to reduce the difference; a detection model determination unit, used to use the trained initial model corresponding to each category as the detection model corresponding to the category; and a model set construction unit, used to construct a model set based on the detection model corresponding to each category.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0147] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] See also Fig.11 , Fig.11 The structural block diagram of a detection system provided by an embodiment of the present application is shown. The detection system 200 may include: a processor 210 and an image acquisition unit 220. The processor 210 is electrically connected to the image acquisition unit 220, and the processor 210 may be used to detect the first image based on the detection method provided by the aforementioned embodiments. Optionally, the detection system 200 may also include a memory.

[0149] Among them, the processor 210 may include one or more processing cores. The processor 210 uses various interfaces and lines to connect the various parts of the entire detection system 200, and executes various functions and processes data of the detection system 200 by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor 210 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 111 can integrate one or more combinations of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and computer programs; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 210, and it can be implemented separately through a communication chip. Specifically, the method described in the above embodiment can be executed by one or more processors 210.

[0150] For some embodiments, the memory may include a random access memory (RAM) or a read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets, or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the detection system 200 during use, etc.

[0151] The image detection unit 220 may be a USB camera.

[0152] See also Fig.12 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0153] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 500 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A detection method, characterized in that: include: When the object to be detected is at a target position corresponding to the target workstation, acquiring a first image including the object to be detected; Acquire a target detection model that matches the target workstation and the object to be detected; The first image is detected by using the target detection model, and when it is detected that the object to be detected meets the target condition, it is determined that the object to be detected has passed the detection.

2. The method according to claim 1, characterized in that The first images are multiple, the object to be detected includes a workstation employee, and when the object to be detected is at a target position corresponding to a target workstation, collecting the first image including the object to be detected includes: Determine a standard operating procedure corresponding to the target workstation; Obtain the standard duration corresponding to the standard operating procedure; Determine the first moment when the workstation employee is at the target position corresponding to the target workstation; From the first moment to the standard time, a plurality of second images including the employee at the workstation are continuously acquired, and each of the second images corresponds to one of the first images.

3. The method according to claim 2, characterized in that The standard operation process corresponds to a plurality of process nodes, and the detecting the first image by the target detection model, and determining that the target object passes the detection when it is detected that the target object meets the target condition, includes: Determining at least one key process node from the plurality of process nodes; Each of the first images is detected based on the first standard image corresponding to each of the key process nodes through the target detection model, and when it is detected that the workstation employee meets the target condition, it is determined that the workstation employee has passed the detection.

4. The method according to claim 3, characterized in that The detecting each of the first images based on the first standard image corresponding to each of the key process nodes by the target detection model, and determining that the workstation employee has passed the detection when it is detected that the workstation employee meets the target condition, includes: Acquire a first image matching a first standard image corresponding to any one of the key process nodes as a third image; Respectively acquiring a first number of the third images and a second number of the key process nodes; When the first number is equal to the second number, determining a first order of the third images based on a sequence of timestamps of the third images, and determining a second order of the third images based on a sequence of occurrence of key process nodes corresponding to the third images; When the first sequence is the same as the second sequence, it is determined that the workstation employee has passed the inspection.

5. The method according to claim 1, characterized in that The object to be detected includes a unit to be detected in the product to be detected, and detecting the first image by the target detection model, and determining that the object to be detected passes the detection when it is detected that the object to be detected meets the target condition, includes: Acquire a second standard image corresponding to the target workstation, wherein the second standard image includes standard content; Detecting, by the target detection model, whether a unit to be detected in the product to be detected in the first image matches the standard content; In the case where the unit to be inspected matches the standard content, it is determined that the unit to be inspected in the product to be inspected passes the inspection.

6. The method according to claim 1, characterized in that The object to be detected includes an operation instruction document, and the first image is detected by the target detection model, and when it is detected that the object to be detected meets the target condition, determining that the object to be detected passes the detection includes: Determining a standard text corresponding to the target workstation; Performing text recognition on the operation instruction document in the first image by using the target detection model to obtain a text to be detected corresponding to the operation instruction document; In the case where it is detected that the text to be detected matches the standard text, it is determined that the operation instruction document passes the detection.

7. The method according to claim 1, characterized in that The acquiring of the detection model corresponding to the target workstation includes: Get the target type of the target station; A detection model corresponding to the target type is obtained from a pre-built model set as the target detection model, wherein the model set includes a plurality of detection models, and each detection model corresponds to a type of workstation.

8. The method according to claim 7, characterized in that Before obtaining a detection model corresponding to the target type from a pre-built model set as the target detection model, the method further includes: Acquire training data corresponding to each category, wherein the training data includes a sample image and a label corresponding to the sample image, and the label includes a standard detection result for indicating whether a sample object in the sample image passes the detection; Based on the sample images corresponding to each category, the initial models corresponding to the category are trained respectively, so as to obtain an initial detection result output by each initial model for indicating whether the sample object passes the detection; Based on the difference between the initial detection result and the standard detection result, training the initial model corresponding to the category to reduce the difference; The initial model corresponding to each category after training is used as the detection model corresponding to the category; A model ensemble is constructed based on the detection model corresponding to each category.

9. A detection device, characterized in that: include: A collection unit, used for collecting a first image including the object to be detected when the object to be detected is at a target position corresponding to the target station; An acquisition unit, used for acquiring a target detection model matching the target station and the object to be detected; The detection unit is used to detect the first image through the target detection model, and determine that the object to be detected has passed the detection when it is detected that the object to be detected meets the target condition.

10. A detection system, characterized in that: The system comprises a processor and an image acquisition unit, wherein the processor is connected to the image acquisition unit; The image acquisition unit is used to acquire a first image including an object to be detected; The processor is used to detect the first image based on the method described in any one of claims 1-8.

Images