Component assembly abnormity detection method

Abstract

The invention discloses a component assembly anomaly detection method, and relates to the technical field of computer vision, and the method comprises the steps: firstly obtaining an actual measurement image of a to-be-detected component, and determining a first character region sub-image in the actual measurement image based on a character region detection model; determining first character group information of each row in the first character region sub-image based on a character recognition model; obtaining pre-stored target unique character group information of a target element corresponding to the to-be-detected element; and if the first character group information of each row contains the target unique character group information, indicating that the unique character group information of the to-be-detected element is the same as the unique character group of the corresponding target element, determining that the to-be-detected element is normally assembled, and otherwise, determining that the to-be-detected element is abnormally assembled. The problem that the accuracy of element assembly anomaly detection is poor due to the character difference in the actual measurement image and the standard image is solved, and therefore the accuracy of element assembly anomaly detection is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a component assembly anomaly detection method. Background Art

[0002] During the component inspection process of automatic optical inspection equipment (AOI), accurate identification of the character group information on the component, especially the unique character group information of the component, is crucial to ensure the correctness of component assembly and avoid misjudgment of wrong materials.

[0003] In the prior art, a standard image and a measured image corresponding to the component to be detected are obtained, and the standard image and the measured image are compared, that is, color analysis or pixel-level matching is used to identify whether there is a difference between the standard image and the measured image. If there is a difference between the standard image and the measured image, it is determined that the component is assembled abnormally. If there is no difference between the standard image and the measured image, it is determined that the component is assembled normally. The problem with the prior art is that for the same component, the characters on the surface of the component will be diverse due to differences in production batches, timestamps, and suppliers. The image comparison method is prone to false alarms for the diversity of characters on the surface of the component. That is, due to the diversity of characters, components that are assembled normally are mistakenly identified as being assembled abnormally, resulting in poor accuracy in detecting component assembly abnormalities. Summary of the invention

[0004] The present application provides a component assembly anomaly detection method to solve the problem of poor accuracy of component assembly anomaly detection in the prior art.

[0005] In a first aspect, the present application provides a method for detecting component assembly anomalies, the method comprising:

[0006] Acquire a measured image of the component to be detected, input the measured image into a character region detection model, and determine a first character region sub-image in the measured image based on the character region detection model;

[0007] Inputting the first character region sub-image into a character recognition model, and determining first character group information of each row in the first character region sub-image based on the character recognition model;

[0008] The pre-saved target unique character group information of the target element corresponding to the element to be detected is obtained. If the first character group information of each row contains the target unique character group information, it is determined that the element to be detected is assembled normally; otherwise, it is determined that the element to be detected is assembled abnormally.

[0009] The above technical solution has the following advantages or beneficial effects:

[0010] Since each component has unique character group information, for the component to be detected, if the unique character group information of the component to be detected is the same as the unique character group of the corresponding target component, it is determined that the assembly of the component to be detected is normal, otherwise it is determined that the assembly of the component to be detected is abnormal. Based on the above considerations, the present application provides a solution for detecting component assembly abnormalities based on unique character group information. First, the measured image of the component to be detected is obtained, and the first character area sub-image in the measured image is determined based on the character area detection model; then, based on the character recognition model, the first character group information of each row in the first character area sub-image is determined; the target unique character group information of the target element corresponding to the component to be detected that is pre-saved is obtained; if the first character group information of each row contains the target unique character group information, it means that the unique character group information of the component to be detected is the same as the unique character group of the corresponding target element, then it is determined that the assembly of the component to be detected is normal, otherwise it is determined that the assembly of the component to be detected is abnormal. Compared with the method based on image comparison, it avoids the problem of poor accuracy of component assembly abnormality detection due to the difference in characters between the measured image and the standard image, thereby improving the accuracy of component assembly abnormality detection.

[0011] In some embodiments of the present application, the process of pre-saving the target unique character group information of the target element includes:

[0012] Acquire a standard image of the target element, input the standard image into the character region detection model, and determine a second character region sub-image in the standard image based on the character region detection model;

[0013] Inputting the second character region sub-image into the character recognition model, and determining second character group information of each row in the second character region sub-image based on the character recognition model;

[0014] Acquire each pre-saved unique character group information; for the second character group information of each row, match the second character group information with each unique character group information, and if the match is successful, determine the second character group information as the target unique character group information of the target element and save it.

[0015] The above technical solution has the following advantages or beneficial effects:

[0016] In order to improve the intelligence and automation of component assembly anomaly detection, the present application provides a method for intelligently determining the target unique character group information of the target component. First, a standard image of the target component is obtained, and the second character area sub-image in the standard image is determined based on the character area detection model; based on the character recognition model, the second character group information of each row in the second character area sub-image is determined; then, each pre-saved unique character group information is obtained, and for each row of the second character group information, the second character group information is matched with each unique character group information. If the match is successful, it means that the second character group information of the row is the unique character group information, and the second character group information of the row is determined as the target unique character group information of the target component and saved. Thus, the intelligence of pre-saving the target unique character group information of the target component is realized.

[0017] In some embodiments of the present application, the obtaining of pre-saved information of each unique character group includes:

[0018] Acquire first text information of each unique character group information stored in advance;

[0019] Matching the second character group information with each of the unique character group information comprises:

[0020] The second text information of the second character group information is matched with the first text information of each unique character group information.

[0021] The above technical solution has the following advantages or beneficial effects:

[0022] Each unique character group information saved in advance includes the first text information of each unique character group information, and the second text information of the second character group information is matched with the first text information of each unique character group information. If the match is successful, the second character group information is determined as the target unique character group information of the target element and saved. Thus, a solution for determining the target unique character group information of the target element based on the text information matching of the unique character group information is realized.

[0023] In some embodiments of the present application, the obtaining of pre-saved information of each unique character group includes:

[0024] Acquire first text information of each unique character group information stored in advance;

[0025] Matching the second character group information with each of the unique character group information comprises:

[0026] The second text information of the second character group information is matched with the first text information of each unique character group information.

[0027] The above technical solution has the following advantages or beneficial effects:

[0028] In some cases, for a new target element, it is possible that the pre-saved unique character group information does not contain the target unique character group information of the target element. The present application takes into account that the regular expression of the unique character group information of the element has certain rules. Therefore, in order to ensure that the target unique character group information of the target element can be determined, the present application pre-saves the first regular expression of each unique character group information. For the second character group information of each row, the second regular expression of the second character group information is matched with the first regular expression of each unique character group information. If the match is successful, the second character group information is determined as the target unique character group information of the target element and saved. Thereby, it is achieved that when the target unique character group information of the target element is not contained in the pre-saved unique character group information, the target unique character group information of the target element can be determined.

[0029] In some embodiments of the present application, if there is no successfully matched second character group information, the method further includes:

[0030] Obtain each pre-saved non-unique character group information, and for each line of second character group information, determine whether the second character group information exists in the each non-unique character group information; if so, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

[0031] The above technical solution has the following advantages or beneficial effects:

[0032] If both the text information and the regular expression rules based on the second character group information of each row fail to match, the present application provides a solution for determining the target unique character group information of the target element based on filtering. That is, obtain each pre-saved non-unique character group information, and for each row of the second character group information, determine whether the second character group information exists in each non-unique character group information. If so, it means that the second character group information of the row is non-unique character group information, and the second character group information of the row is filtered out. Because there is unique character group information in the target element, there will be retained second character group information, and the retained second character group information is determined as the target unique character group information. Thus, a solution for determining the target unique character group information of the target element based on filtering is realized.

[0033] In some embodiments of the present application, if there is no successfully matched second character group information, the method further includes:

[0034] A third regular expression for obtaining pre-saved information of each non-unique character group;

[0035] For the second character group information of each row, determine whether the second regular expression of the second character group information exists in the third regular expression of each non-unique character group information; if so, filter out the second character group information corresponding to the second regular expression, and determine the retained second character group information as the target unique character group information.

[0036] The above technical solution has the following advantages or beneficial effects:

[0037] If both the text information and the regular expression rules based on the second character group information of each row fail to match, the present application provides a solution for determining the target unique character group information of the target element based on filtering. That is, the third regular expression of each pre-saved non-unique character group information is obtained, and for each row of the second character group information, it is determined whether the second regular expression of the second character group information exists in each non-unique character group information. If so, it means that the second character group information of the row is non-unique character group information, and the second character group information of the row is filtered out. Because there is unique character group information in the target element, there will be retained second character group information, and the retained second character group information is determined as the target unique character group information. Thus, a solution for determining the target unique character group information of the target element based on filtering is realized.

[0038] In some embodiments of the present application, if there is no successfully matched second character group information, the method further includes:

[0039] Inputting the standard image into a component category detection model, and determining target category information of the target component in the standard image based on the component category detection model;

[0040] Determine a target character length range corresponding to the target category information according to the character length ranges corresponding to the respective category information stored in advance;

[0041] For the second character group information of each row, determine the character length of the second character group information; judge whether the character length belongs to the target character length range, if not, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

[0042] The above technical solution has the following advantages or beneficial effects:

[0043] If both the text information and the regular expression rules based on the second character group information of each line fail to match, the present application provides a scheme for determining the target unique character group information of the target element based on filtering. Considering that the character length of the unique character group information of elements of different categories is within a certain range, if it exceeds the range, it means that it is not the unique character group information. Based on the above considerations, the present application determines the target category information of the target element in the standard image based on the element category detection model, and then determines the target character length range corresponding to the target category information according to the character length ranges corresponding to each pre-saved category information; filters out the second character group information whose character length does not belong to the character length range, and determines the retained second character group information as the target unique character group information. Thereby, a scheme for determining the target unique character group information of the target element based on filtering is realized.

[0044] In some embodiments of the present application, determining the retained second character group information as the target unique character group information includes:

[0045] If only one row of second character group information is retained, determining the second character group information of the row as the target unique character group information;

[0046] If multiple lines of second character group information are retained, the method further includes:

[0047] Inputting the second character group information of each retained line into a character information classification model respectively, and determining character category information of the second character group information of each line based on the character information classification model; wherein the character category information includes unique category information and non-unique category information;

[0048] The second character group information whose character category information is unique type information is determined as the target unique character group information.

[0049] The above technical solution has the following advantages or beneficial effects:

[0050] In order to make the determined target unique character group information more accurate, after filtering out the second character group information, determine whether there is only one line of second character group information left. Because the unique character group information of the element has only one line, if only one line of second character group information is retained, the second character group information of that line is determined as the target unique character group information, which can ensure that the determined target unique character group information is accurate. If multiple lines of second character group information are retained, it means that at least one line of second character group information is misidentified as unique character group information. At this time, the character category information of each retained line of second character group information is determined based on the character information classification model; the second character group information whose character category information is unique information is determined as the target unique character group information. This ensures the accuracy of the determined target unique character group information.

[0051] In some embodiments of the present application, the step of inputting the retained second character group information of each line into a character information classification model, and determining the character category information of the second character group information of each line based on the character information classification model comprises:

[0052] For each retained line of second character group information, determining a line number corresponding to the second character group information; inputting the second character group information and the corresponding line number into the character information classification model, and determining character category information of the second character group information based on the character information classification model;

[0053] Among them, the corresponding weight parameter is determined according to the row number corresponding to the sample character group information in the training set, and the confidence that the character category information of the sample character group information is the only category information is predicted according to the weight parameter; and the character information classification model is trained according to the confidence and the character category information label corresponding to the sample character group information.

[0054] The above technical solution has the following advantages or beneficial effects:

[0055] In order to make the character category information of the second character group information of each row more accurate, the present application takes into account that there is a certain regularity in the row where the unique character group information of the element is located. For example, the unique character group information of a certain type of element is in the first two rows, and the unique character group information of another type of element is in the last two rows, and so on. Therefore, the present application inputs the second character group information and the corresponding row number into the character information classification model for each retained row of the second character group information, and determines the character category information of the second character group information based on the character information classification model. Moreover, when training the character information classification model, corresponding weight parameters are assigned to different row numbers. Optionally, the weight parameter corresponding to the row number with a higher probability of the unique character group information appearing in the row is also larger. In this way, the character category information of the second character group information is more accurately determined by combining the second character group information and the corresponding row number.

[0056] In some embodiments of the present application, before determining that the component to be detected is assembled abnormally, the method further includes:

[0057] Determine at least one target replacement unique character group information corresponding to the target unique character group information according to at least one replacement unique character group information corresponding to each pre-stored unique character group information;

[0058] If the first character group information of each row and the at least one target replacement unique character group information do not have the same character group information, it is determined that the component to be detected is assembled abnormally;

[0059] The method further comprises:

[0060] If the first character group information of each row and the at least one target replacement unique character group information contain the same character group information, it is determined that the component to be inspected is assembled normally.

[0061] The above technical solution has the following advantages or beneficial effects:

[0062] The present application takes into account that there may be at least one replacement element for the target element, and at least one replacement unique character group information can be determined and saved based on the at least one replacement element. If the first character group information of each row does not contain the target unique character group information, it is possible that a replacement element of the target element is used. Therefore, if the first character group information of each row and at least one target replacement unique character group information have the same character group information, it is determined that the replacement element is used and the assembly of the component to be detected is normal; if the first character group information of each row and at least one target replacement unique character group information do not have the same character group information, it is determined that the replacement element is not used and the assembly of the component to be detected is abnormal.

[0063] In a second aspect, the present application provides a component assembly abnormality detection device, the device comprising:

[0064] A first determination module is used to obtain a measured image of the component to be detected, input the measured image into a character area detection model, and determine a first character area sub-image in the measured image based on the character area detection model;

[0065] A second determination module, configured to input the first character region sub-image into a character recognition model, and determine first character group information of each row in the first character region sub-image based on the character recognition model;

[0066] The detection module is used to obtain the pre-saved target unique character group information of the target element corresponding to the element to be detected. If the first character group information of each row contains the target unique character group information, it is determined that the element to be detected is assembled normally; otherwise, it is determined that the element to be detected is assembled abnormally.

[0067] In a third aspect, the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0068] Memory, used to store computer programs;

[0069] The processor is used to implement the component assembly abnormality detection method when executing the program stored in the memory.

[0070] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the component assembly anomaly detection method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0072] Figure 1 A schematic diagram of the component assembly anomaly detection process provided by this application;

[0073] Figure 2 A schematic diagram of a first process for determining target unique character group information of a target element provided by the present application;

[0074] Figure 3 A schematic diagram of a second process for determining target unique character group information of a target element provided by the present application;

[0075] Figure 4 A schematic diagram of a third process for determining target unique character group information of a target element provided by the present application;

[0076] Figure 5 Schematic diagram of the standard image and measured image provided for this application;

[0077] Figure 6 A schematic diagram for determining character group information in a measured image and a standard image provided by the present application;

[0078] Figure 7 A schematic diagram of the character direction rotation correction provided in this application;

[0079] Figure 8 A detailed flow chart of the target unique character group information of the target element provided by this application;

[0080] Fig. 9 A schematic diagram of the row number of the character group information in the element provided by this application;

[0081] Fig.10 The front-end page provided for this application displays a schematic diagram of the main material number determined;

[0082] Fig.11 This is a schematic diagram of the structure of the component assembly abnormality detection device provided in this application;

[0083] Fig.12This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0084] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0085] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0086] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0087] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0088] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0090] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0091] During the component inspection process, the automatic optical inspection (AOI) equipment accurately identifies the silk-screen information on the component (the silk-screen information includes character information, and all character information contained in a line on the component is called the character group information of a line), especially the main material number (the unique character group information of the component), which is crucial to ensure the correctness of assembly and avoid misjudgment of wrong materials. Traditional detection methods rely on direct image comparison, that is, identifying differences through color analysis or pixel-level matching. Considering that the production batch characters, timestamp characters, supplier characters, etc. of components of the same model may also be different, traditional detection methods are incapable of dealing with the diversity of character formats caused by production batches, timestamps, and suppliers, and are prone to false alarms. Especially in the context of rapid changes in the production environment and diversified suppliers, the accuracy of component assembly anomaly detection is difficult to guarantee.

[0092] This application proposes a component assembly anomaly detection method, which aims to extract the unique character group information (main material number) of the component core from complex multi-line character group information with high precision. Focus on identifying the unique character group information of the component, rather than getting entangled in volatile non-unique character group information, such as production batches, timestamps, suppliers, etc. This application can ensure the accuracy of judgment, reduce false alarms of wrong materials, and lay a solid foundation for subsequent polarity checks or wrong material verification. Optimizing the accuracy and adaptability of the AOI detection process effectively responds to the diverse challenges faced by the modern electronic manufacturing field.

[0093] Figure 1 The schematic diagram of the component assembly anomaly detection process provided in this application includes the following steps:

[0094] S101: Acquire a measured image of a component to be detected, input the measured image into a character region detection model, and determine a first character region sub-image in the measured image based on the character region detection model;

[0095] S102: inputting the first character region sub-image into a character recognition model, and determining first character group information of each row in the first character region sub-image based on the character recognition model;

[0096] S103: Acquire pre-saved target unique character group information of the target element corresponding to the element to be detected. If the first character group information of each row contains the target unique character group information, determine that the element to be detected is assembled normally; otherwise, determine that the element to be detected is assembled abnormally.

[0097] The component assembly anomaly detection method provided in the present application is applied to an electronic device, which may be a PC, a computer or other device, or a server.

[0098] The electronic device stores a trained character area detection model, and the character area detection model is used to detect the area where the characters in the input component image are located. After obtaining the measured image of the component to be detected, the measured image is input into the character area detection model, and the first character area sub-image in the measured image is determined based on the character area detection model. The training process of the character area detection model is as follows: the sample component image and the character area sub-image label where the character group in the sample component image is located are input into the character area detection model to be trained, the character area sub-image in the sample component image is predicted based on the character area detection model to be trained, the loss value is determined based on the prediction result and the label, and the character area detection model is iteratively trained based on the loss value. Optionally, when the number of iterations reaches a set number of thresholds, it is determined that the character area detection model training is completed, or when the loss value during the iteration process meets the requirements, it is determined that the character area detection model training is completed. It should be noted that if the character area sub-image annotated in the sample component image during the training of the character area detection model is the character area sub-image of each row, that is, each row of character group information corresponds to a character area sub-image, then the first character area sub-image in the measured image determined based on the character area detection model is also a first character area sub-image corresponding to each row. If the character area sub-image annotated in the sample component image during the training of the character area detection model is a character area sub-image containing all character groups, then the first character area sub-image in the measured image determined based on the character area detection model is also a first character area sub-image.

[0099] The electronic device pre-stores a trained character recognition model, and the character recognition model is used to recognize the character group information of each line in the input character area image. After determining the first character area sub-image, the first character area sub-image is input into the character recognition model, and based on the character recognition model, the first character group information of each line in the first character area sub-image is determined. The training process of the character recognition model is as follows: the sample character area image and the label of the character group information of each line in the sample character area image and the line number label of each line are input into the character recognition model to be trained, and the first character group information of each line and the line number of each line in the sample character area image are predicted based on the character recognition model to be trained; the first loss value is determined according to the predicted first character group information of each line and the label of the first character group information of each line; the second loss value is determined according to the predicted line number of each line and the label of the line number of each line; the loss value of the character recognition model can be obtained by weighted summing the first loss value and the second loss value. The character recognition model is iteratively trained according to the loss value. Optionally, when the number of iterations reaches a set threshold, it is determined that the character recognition model training is completed, or when the loss value during the iteration process meets the requirements, it is determined that the character recognition model training is completed.

[0100] In addition, after determining the first character region sub-image in the measured image and before inputting the first character region sub-image into the character recognition model, the method further includes:

[0101] The first character region sub-image is input into the character direction classification model, and the character direction corresponding to the first character region sub-image is determined based on the character direction classification model; if the character direction is different from a preset target character direction, the first character region sub-image is rotated according to the target character direction. The target character direction is, for example, the character direction of the forward direction of the character, that is, the direction in which the characters are arranged from left to right according to the semantic information.

[0102] The training process of the character direction classification model is as follows: input the sample character area image and the character direction label of the sample character area image into the character direction classification model to be trained, and predict the character direction of the sample character area image based on the character direction classification model to be trained; determine the loss value of the character direction classification model according to the predicted character direction and the character direction label. Iterate the training of the character direction classification model according to the loss value. Optionally, when the number of iterations reaches a set threshold, it is determined that the training of the character direction classification model is completed, or when the loss value during the iteration process meets the requirements, it is determined that the training of the character direction classification model is completed.

[0103] There is a corresponding standard image for the measured image of the component to be detected, and the component in the standard image is the target component. The electronic device obtains the pre-saved target unique character group information of the target component corresponding to the component to be detected. If the first character group information of each row contains the target unique character group information, it is determined that the component to be detected is assembled normally; otherwise, it is determined that the component to be detected is assembled abnormally.

[0104] Since each component has unique character group information, for the component to be detected, if the unique character group information of the component to be detected is the same as the unique character group of the corresponding target component, it is determined that the assembly of the component to be detected is normal, otherwise it is determined that the assembly of the component to be detected is abnormal. Based on the above considerations, the present application provides a solution for detecting component assembly abnormalities based on unique character group information. First, the measured image of the component to be detected is obtained, and the first character area sub-image in the measured image is determined based on the character area detection model; then, based on the character recognition model, the first character group information of each row in the first character area sub-image is determined; the target unique character group information of the target element corresponding to the component to be detected that is pre-saved is obtained; if the first character group information of each row contains the target unique character group information, it means that the unique character group information of the component to be detected is the same as the unique character group of the corresponding target element, then it is determined that the assembly of the component to be detected is normal, otherwise it is determined that the assembly of the component to be detected is abnormal. Compared with the method based on image comparison, it avoids the problem of poor accuracy of component assembly abnormality detection due to the difference in characters between the measured image and the standard image, thereby improving the accuracy of component assembly abnormality detection.

[0105] Figure 2 The first process diagram for determining target unique character group information of a target element provided in the present application includes the following steps:

[0106] S201: Acquire a standard image of a target component, input the standard image into the character region detection model, and determine a second character region sub-image in the standard image based on the character region detection model;

[0107] S202: inputting the second character region sub-image into the character recognition model, and determining second character group information of each row in the second character region sub-image based on the character recognition model;

[0108] S203: Acquire each pre-saved unique character group information; for the second character group information of each row, match the second character group information with each unique character group information, and if the match is successful, determine the second character group information as the target unique character group information of the target element and save it.

[0109] In the present application, a standard image of a target element corresponding to the element to be detected is obtained, the standard image is input into a character area detection model, and a second character area sub-image in the standard image is determined based on the character area detection model; wherein, the process of determining the second character area sub-image in the standard image based on the character area detection model is similar to the process of determining the first character area sub-image in the measured image based on the character area detection model, and will not be described in detail here. The second character area sub-image is input into a character recognition model, and the second character group information of each row in the second character area sub-image is determined based on the character recognition model; wherein, the process of determining the second character group information of each row in the second character area sub-image based on the character recognition model is similar to the process of determining the first character group information of each row in the first character area sub-image based on the character recognition model, and will not be described in detail here.

[0110] The electronic device is pre-stored with a main material number library, which contains the main material number of each component, that is, the unique character group information of each component. That is, the electronic device is pre-stored with each unique character group information. For the second character group information of each row determined, the second character group information is matched with each unique character group information. If the match is successful, the second character group information is determined as the target unique character group information of the target component and saved. Optionally, the second character group information is matched with each unique character group information. If the second character group information exists in each unique character group information, it means that the match is successful, otherwise it is determined that the match fails.

[0111] In order to improve the intelligence and automation of component assembly anomaly detection, the present application provides a method for intelligently determining the target unique character group information of the target component. First, a standard image of the target component is obtained, and the second character area sub-image in the standard image is determined based on the character area detection model; based on the character recognition model, the second character group information of each row in the second character area sub-image is determined; then, each pre-saved unique character group information is obtained, and for each row of the second character group information, the second character group information is matched with each unique character group information. If the match is successful, it means that the second character group information of the row is the unique character group information, and the second character group information of the row is determined as the target unique character group information of the target component and saved. Thus, the intelligence of pre-saving the target unique character group information of the target component is realized.

[0112] Figure 3 A schematic diagram of a second process for determining target unique character group information of a target element provided in the present application includes the following steps:

[0113] S301: Acquire a standard image of a target component, input the standard image into the character region detection model, and determine a second character region sub-image in the standard image based on the character region detection model;

[0114] S302: inputting the second character region sub-image into the character recognition model, and determining second character group information of each row in the second character region sub-image based on the character recognition model;

[0115] S303: Acquire the first text information of each unique character group information saved in advance; for the second character group information of each line, match the second text information of the second character group information with the first text information of each unique character group information; if the match is successful, determine the second character group information as the target unique character group information of the target element and save it.

[0116] Each unique character group information saved in advance includes the first text information of each unique character group information, and the second text information of the second character group information is matched with the first text information of each unique character group information. If the match is successful, the second character group information is determined as the target unique character group information of the target element and saved. Optionally, the second text information of the second character group information is matched with the first text information of each unique character group information. If the second text information of the second character group information exists in the first text information of each unique character group information, it means that the match is successful, otherwise it is determined that the match fails. Thus, a solution for determining the target unique character group information of the target element based on text information matching of the unique character group information is realized.

[0117] Figure 4The third process diagram for determining target unique character group information of a target element provided by the present application includes the following steps:

[0118] S401: Acquire a standard image of a target component, input the standard image into the character region detection model, and determine a second character region sub-image in the standard image based on the character region detection model;

[0119] S402: inputting the second character region sub-image into the character recognition model, and determining second character group information of each row in the second character region sub-image based on the character recognition model;

[0120] S403: Obtain the first regular expression of each unique character group information saved in advance; for the second character group information of each line, match the second regular expression of the second character group information with the first regular expression of each unique character group information, and if the match is successful, determine the second character group information as the target unique character group information of the target element and save it.

[0121] In some cases, for a new target element, it is possible that the pre-saved unique character group information does not contain the target unique character group information of the target element. The present application takes into account that the regular expression of the unique character group information of the element has certain rules. Therefore, in order to ensure that the target unique character group information of the target element can be determined, the present application pre-saves the first regular expression of each unique character group information. For the second character group information of each row, the second regular expression of the second character group information is matched with the first regular expression of each unique character group information. If the match is successful, the second character group information is determined as the target unique character group information of the target element and saved. Optionally, the second regular expression of the second character group information is matched with the first regular expression of each unique character group information. If the second regular expression of the second character group information exists in the first regular expression of each unique character group information, it means that the match is successful, otherwise it is determined that the match fails. Thereby, it is achieved that when the target unique character group information of the target element is not contained in the pre-saved unique character group information, the target unique character group information of the target element can be determined.

[0122] In some embodiments of the present application, if there is no successfully matched second character group information, the method further includes:

[0123] Obtain each pre-saved non-unique character group information, and for each line of second character group information, determine whether the second character group information exists in the each non-unique character group information; if so, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

[0124] If both the text information and the regular expression rules based on the second character group information of each row fail to match, the present application provides a solution for determining the target unique character group information of the target element by filtering. A non-main material number library is pre-stored in the electronic device, and the non-main material number library contains the non-main material numbers of each component, that is, the non-unique character group information of each component. That is, each non-unique character group information is pre-stored in the electronic device. For each row of second character group information determined, it is determined whether the second character group information exists in each non-unique character group information. If so, it means that the second character group information of the row is non-unique character group information, and the second character group information of the row is filtered out. Because there is unique character group information in the target element, there will be retained second character group information, and the retained second character group information is determined as the target unique character group information. Thus, a solution for determining the target unique character group information of the target element based on filtering is realized.

[0125] In some embodiments of the present application, if there is no successfully matched second character group information, the method further includes:

[0126] A third regular expression for obtaining pre-saved information of each non-unique character group;

[0127] For the second character group information of each row, determine whether the second regular expression of the second character group information exists in the third regular expression of each non-unique character group information; if so, filter out the second character group information corresponding to the second regular expression, and determine the retained second character group information as the target unique character group information.

[0128] If both the text information and the regular expression rules based on the second character group information of each row fail to match, the present application provides a solution for determining the target unique character group information of the target element based on filtering. That is, the third regular expression of each pre-saved non-unique character group information is obtained, and for each row of the second character group information, it is determined whether the second regular expression of the second character group information exists in each non-unique character group information. If so, it means that the second character group information of the row is non-unique character group information, and the second character group information of the row is filtered out. Because there is unique character group information in the target element, there will be retained second character group information, and the retained second character group information is determined as the target unique character group information. Thus, a solution for determining the target unique character group information of the target element based on filtering is realized.

[0129] In some embodiments of the present application, if there is no successfully matched second character group information, the method further includes:

[0130] Inputting the standard image into a component category detection model, and determining target category information of the target component in the standard image based on the component category detection model;

[0131] Determine a target character length range corresponding to the target category information according to the character length ranges corresponding to the respective category information stored in advance;

[0132] For the second character group information of each row, determine the character length of the second character group information; judge whether the character length belongs to the target character length range, if not, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

[0133] A trained component category detection model is stored in the electronic device, and the component category detection model is used to detect the category information of the component in the input component image. The category information of the component is, for example, capacitance, resistance, inductance, etc. The training process of the component category detection model is as follows: the sample component image and the component category label in the sample component image are input into the component category detection model to be trained, the component category in the sample component image is predicted based on the component category detection model to be trained, the loss value is determined according to the predicted component category and the component category label, and the component category detection model is iteratively trained according to the loss value. Optionally, when the number of iterations reaches a set number threshold, it is determined that the training of the component category detection model is completed, or when the loss value during the iteration process meets the requirements, it is determined that the training of the component category detection model is completed.

[0134] If both the text information and the regular expression rules based on the second character group information of each line fail to match, the present application provides a scheme for determining the target unique character group information of the target element based on filtering. Considering that the character length of the unique character group information of elements of different categories is within a certain range, if it exceeds the range, it means that it is not the unique character group information. Based on the above considerations, the present application determines the target category information of the target element in the standard image based on the element category detection model, and then determines the target character length range corresponding to the target category information according to the character length ranges corresponding to each pre-saved category information; filters out the second character group information whose character length does not belong to the character length range, and determines the retained second character group information as the target unique character group information. Thereby, a scheme for determining the target unique character group information of the target element based on filtering is realized.

[0135] In some embodiments of the present application, determining the retained second character group information as the target unique character group information includes:

[0136] If only one row of second character group information is retained, determining the second character group information of the row as the target unique character group information;

[0137] If multiple lines of second character group information are retained, the method further includes:

[0138] Inputting the second character group information of each retained line into a character information classification model respectively, and determining character category information of the second character group information of each line based on the character information classification model; wherein the character category information includes unique category information and non-unique category information;

[0139] The second character group information whose character category information is unique type information is determined as the target unique character group information.

[0140] In order to make the determined target unique character group information more accurate, after filtering out the second character group information, it is determined whether only one row of second character group information remains. Because the unique character group information of the component has only one row, if only one row of second character group information is retained, the second character group information of that row is determined as the target unique character group information, which can ensure that the determined target unique character group information is accurate.

[0141] If multiple lines of second character group information are retained, it means that at least one line of second character group information is mistakenly identified as unique character group information. At this time, the character category information of each retained line of second character group information is determined based on the character information classification model; the second character group information whose character category information is unique class information is determined as the target unique character group information. Thereby, the accuracy of the determined target unique character group information is ensured. The training process of the character information classification model is as follows: the sample character group information and the character category label are input into the character information classification model to be trained, and the character category of the sample character group information is predicted based on the character information classification model to be trained; the loss value is determined according to the predicted character category and character category label, and the character information category model is iteratively trained according to the loss value. Optionally, when the number of iterations reaches the set number threshold, it is determined that the character information category model training is completed, or when the loss value meets the requirements during the iteration process, it is determined that the character information category model training is completed.

[0142] In some embodiments of the present application, the step of inputting the retained second character group information of each line into a character information classification model, and determining the character category information of the second character group information of each line based on the character information classification model comprises:

[0143] For each retained line of second character group information, determining a line number corresponding to the second character group information; inputting the second character group information and the corresponding line number into the character information classification model, and determining character category information of the second character group information based on the character information classification model;

[0144] Among them, the corresponding weight parameter is determined according to the row number corresponding to the sample character group information in the training set, and the confidence that the character category information of the sample character group information is the only category information is predicted according to the weight parameter; and the character information classification model is trained according to the confidence and the character category information label corresponding to the sample character group information.

[0145] In order to make the character category information of the second character group information of each row more accurate, the present application takes into account that there is a certain regularity in the row where the unique character group information of the element is located. For example, the unique character group information of a certain type of element is in the first two rows, and the unique character group information of another type of element is in the last two rows, and so on. Therefore, the present application inputs the second character group information and the corresponding row number into the character information classification model for each retained row of the second character group information, and determines the character category information of the second character group information based on the character information classification model. Moreover, when training the character information classification model, corresponding weight parameters are assigned to different row numbers. Optionally, the weight parameter corresponding to the row number with a higher probability of the unique character group information appearing in the row is also larger. In this way, the character category information of the second character group information is more accurately determined by combining the second character group information and the corresponding row number.

[0146] In an embodiment of the present application, the training process of the character information classification model is as follows: the sample character group information, the row number corresponding to the sample character group information, and the character category label are input into the character information classification model to be trained. The row number refers to the row number of the sample character group information in the sample element. For example, the row number corresponding to the sample character group information in the first row of the sample element is 1. Predict the character category of the sample character group information based on the character information classification model to be trained; determine the loss value based on the predicted character category and the character category label, and iteratively train the character information category model based on the loss value. Optionally, when the number of iterations reaches a set number threshold, it is determined that the character information category model training is completed, or when the loss value meets the requirements during the iteration process, it is determined that the character information category model training is completed.

[0147] Among them, the corresponding weight parameter is determined according to the row number corresponding to the sample character group information in the training set. For example, for some components, their unique character group information is in the first two rows, then the weight parameters corresponding to the row numbers of the first two rows are larger, and the weight parameters corresponding to the remaining row numbers are smaller. For another example, for other components, the unique character group information is in the last two rows, then the weight parameters corresponding to the row numbers of the last two rows are larger, and the weight parameters corresponding to the remaining row numbers are smaller.

[0148] In some embodiments of the present application, before determining that the component to be detected is assembled abnormally, the method further includes:

[0149] Determine at least one target replacement unique character group information corresponding to the target unique character group information according to at least one replacement unique character group information corresponding to each pre-stored unique character group information;

[0150] If the first character group information of each row and the at least one target replacement unique character group information do not have the same character group information, it is determined that the component to be detected is assembled abnormally;

[0151] The method further comprises:

[0152] If the first character group information of each row and the at least one target replacement unique character group information contain the same character group information, it is determined that the component to be inspected is assembled normally.

[0153] The present application takes into account that there may be at least one replacement element for the target element, and at least one replacement unique character group information can be determined and saved based on the at least one replacement element. The electronic device can pre-save a replacement library, which includes replacement unique character group information of all replacement elements corresponding to the target element. If the first character group information of each row does not contain the target unique character group information, it is possible that a replacement element of the target element is used. Therefore, if the first character group information of each row and at least one target replacement unique character group information have the same character group information, that is, there is a first character group information in the first character group information of each row that is the same as a certain target replacement unique character group information in at least one target replacement unique character group information. At this time, it is determined that the replacement element is used, and it is determined that the assembly of the component to be detected is normal; if the first character group information of each row and at least one target replacement unique character group information do not have the same character group information, it is determined that the replacement element is not used, and it is determined that the assembly of the component to be detected is abnormal.

[0154] The present application provides a method for detecting component assembly anomalies, which belongs to the field of computer vision in trusted AI. First, the position and content of the multi-line silk screen on the surface of the standard image component are detected by OCR. 1. Preliminary screening: search for the main material number through the main material number library built in advance; 2. Preliminary elimination: delete the brand name, manufacturer and other fixed characters on the surface of the standard image through the non-main material number library built in advance. Delete characters whose length is not within the preset character length range; 3. Filter using regular expressions: use common prefixes and suffixes such as date and batch symbols to filter and remove unimportant information such as date and batch; 4. The machine learning algorithm classifies the remaining characters into two categories (0: does not belong to the main material number, 1: belongs to the main material number); wherein the weight is set by the character content and line number. Based on the extracted main material number, the component is screened to see if it is the wrong material.

[0155] The technical solution of this application has the following advantages or beneficial effects:

[0156] Trustworthy features:

[0157] Real-time: This application makes subsequent logical judgments based on the inference results of the deep learning model. Each model can process the images to be inferred from the AOI device to the algorithm within a given time period, which meets the real-time characteristics of the trustworthy feature.

[0158] Reproducibility: The same image can be transmitted multiple times to the same batch of models for inference. The same results can be obtained by inference multiple times in different operating environments (different platforms, different inference times, and different times), which meets the reproducibility feature of the trustworthiness feature.

[0159] Controllability: This application can be turned on or off by the user at any time during use. The importance of the inference results and the degree of adoption can be intervened by the user, which conforms to the controllability feature of the trustworthy characteristic.

[0160] Figure 5 Schematic diagram of the standard image and measured image provided for this application, Figure 5 The middle left image is the standard image. Figure 5 The middle right picture is a measured image. The image contains five lines of character group information, of which the first line is unique character group information and the remaining lines are non-unique character group information.

[0161] The component assembly anomaly detection process provided by the present application is described in detail below in conjunction with the accompanying drawings. For ease of understanding, the unique character group information is described below as the main material number; the non-unique character group information is described as the non-main material number; and the replacement unique character group information is described as the replacement material number.

[0162] Extract the main material number of the standard image, search the substitute material library for the substitute material number corresponding to the main material number, traverse the character group information in the measured image to find out whether there is information related to the main material number or the substitute material number. If so, it passes. If not, it means that the wrong material has occurred and an exception is reported.

[0163] Figure 6 A schematic diagram for determining character group information in a measured image and a standard image provided by the present application, such as Figure 6 As shown, the measured image is input into the character region detection model to obtain the character region sub-image; the character region sub-image is input into the character direction classification model, and after rotation, the positive character region sub-image is obtained; the positive character region sub-image is input into the character recognition model to obtain the first character group information of each line of the measured image. The standard image is input into the character region detection model to obtain the character region sub-image; the character region sub-image is input into the character direction classification model, and after rotation, the positive character region sub-image is obtained; the positive character region sub-image is input into the character recognition model to obtain the second character group information of each line of the standard image.

[0164] like Figure 6 As shown, the character positions and character contents on the standard image and the measured image are obtained: first, the character area detection model is used to detect the character area sub-image on the surface of the component; the character orientation is classified and rotated to be correct through the character direction classification model; the rotated correct character area sub-image is sent to the character recognition model to recognize the character content. The character recognition model can be an OCR character recognition algorithm.

[0165] Figure 7 This is a schematic diagram of the character direction rotation correction provided in this application. Figure 7 The middle left picture is the schematic diagram before rotation. Figure 7 The middle right picture is a schematic diagram after rotation.

[0166] For known components, by querying the BOM, the main material number library, non-main material number library and substitute material library covering the existing components are built offline; the main material number library is a list composed of the character contents of all the main material numbers on the surface of the component, and each main material number is one of the items in the list; the non-main material number library is a list composed of the character contents of all non-main material numbers extracted from the surface of the component, and each line is an item; a mapping relationship between main material numbers that can replace each other is built, and main material numbers that can replace each other are called each other's substitute materials.

[0167] For each rough classification, such as capacitors, inductors, chips and other components, the character length of the main material number is counted separately, and the character length of the main material number for this rough classification is calculated. Assuming that the minimum value of the main material number length counted under a rough classification is n and the maximum value is m, the character length range of this rough classification can be set to: [n-1, m+1].

[0168] Figure 8 The detailed flow chart of the target unique character group information of the target element provided in this application includes the following steps:

[0169] S501: Acquire a standard image of a target component, input the standard image into the character region detection model, and determine a second character region sub-image in the standard image based on the character region detection model;

[0170] S502: inputting the second character region sub-image into the character recognition model, and determining second character group information of each row in the second character region sub-image based on the character recognition model;

[0171] S503: Acquire each pre-saved unique character group information; for each row of second character group information, match the second character group information with each unique character group information, if the match is successful, proceed to S504, if the match fails, proceed to S505;

[0172] S504: Determine the second character group information as the target unique character group information of the target element and save it;

[0173] S505: Acquire each pre-saved non-unique character group information, and for each row of second character group information, if the second character group information exists in each non-unique character group information, filter out the second character group information;

[0174] S506: Obtain a third regular expression for each pre-saved non-unique character group information; for each line of second character group information, if the third regular expression for each non-unique character group information contains a second regular expression for the second character group information, filter out the second character group information corresponding to the second regular expression;

[0175] S507: Input the standard image into a component category detection model, determine the target category information of the target component in the standard image based on the component category detection model; determine the target character length range corresponding to the target category information according to the character length ranges corresponding to the respective category information stored in advance; determine the character length of the second character group information for each line; if the character length does not belong to the target character length range, filter out the second character group information;

[0176] S508: If only one line of second character group information is retained, the second character group information of this line is determined as the target unique character group information; if multiple lines of second character group information are retained, the second character group information of each retained line is input into the character information classification model respectively, and the character category information of the second character group information of each line is determined based on the character information classification model; the second character group information whose character category information is unique information is determined as the target unique character group information.

[0177] Based on the content of each line of characters on the surface of the component in the extracted standard image, the main material number library is first traversed and searched. If a line is found in the main material number library, the content of the line is used as the main material number of the current standard image, recorded as the main material number a.

[0178] All substitute materials corresponding to the main material number a are extracted from the substitute material library. The temporary library constructed by the main material number a and its corresponding substitute materials is called library A. Then the character content in the measured image is extracted, and each line in the measured image is traversed. If the content of library A exists, it means that the measured image and the standard image can be replaced by each other; the measured image detection is normal.

[0179] If all the lines in the main material number are not in the main material number library, it means that the current material is new and the main material number cannot be retrieved through the previously maintained main material number library. At this time, through the previously maintained non-main material number library (the information library maintained by the lines of previously known components that are not main material numbers), the brand, manufacturer and other characters (for example: Hiview, MEDIATEK) in the current component are filtered out. The character lengths of the main material numbers of different roughly classified components are inconsistent. The character line content whose character length is not within the corresponding character length range is filtered out.

[0180] Use regular expressions to further filter, with the help of common prefixes and suffixes such as date and batch symbols, filter out non-main material number information that may change, such as date and batch; extract the naming rules of main material numbers of different rough categories, and use regular expressions to further filter the main material numbers. For example, the main material number of chips is MT{number*4}{letter*2}N.

[0181] After the above screening, if only one line is left, then this line is the main material number. If multiple lines are left, the remaining characters are converted into a numerical vector format that can be recognized by the algorithm through hot single encoding, with each line as a unit. The category to which the component belongs is further classified and estimated through the character information classification model (for example: logistic regression), and the confidence Where C is a preset parameter, z is a polynomial parameter to be trained, and the direction of manual character reading is taken as the positive direction to determine the row number of each row of characters. Because according to previous statistical rules, the main material number is most likely to be in the first two rows, C = 1.2 for the first and second rows, C = 0.8 for the third and fourth rows, and C = 0.5 for the fifth and subsequent rows. If the confidence of the classification result is greater than the given threshold, the classification result is used as the main material number. Otherwise, the character content of all rows obtained after screening is used as the main material number, that is, the measured image is considered to be detected normally only when the measured image successfully matches each main material number. Fig. 9 This is a schematic diagram of the row number of the character group information in the component provided in this application.

[0182] Optionally, if the highest confidence determined by the character information classification model is less than a given threshold, then if the result with the highest confidence ranking is used as the main material number, the advantage is that the complexity of subsequent comparison between the standard image and the measured image can be shortened, but the disadvantage is that an inaccurate result may be obtained for subsequent matching;

[0183] Optionally, if the highest confidence determined by the character information classification model is less than a given threshold, then if the character contents of all rows obtained after screening are used as the main material number to match the measured image, the disadvantage is that it will increase the complexity of subsequent matching, because multiple rows need to be matched, and the multiple rows are mixed with non-main material numbers that may change, which will increase the risk of over-killing; the advantage is that it can reduce errors in judging the main material number and reduce the occurrence of missed detections due to inaccurate judgment of the main material number.

[0184] It should be noted that the result of the main material number confirmation can be displayed on the front-end page. If the personnel do not operate, it will pass. If there is a situation that is inconsistent with previous experience, the personnel can delete the main material number recognition result (select + Delete), and the material number will still be matched in the way of all comparisons. If the personnel do not operate, the main material number located by the algorithm of this application will be matched. Fig.10 The front-end page provided for this application shows the main material number diagram determined, Fig.10 The main material number selected in the middle box is the main material number determined by this application. If the personnel determines that the main material number is incorrect, they can select the Delete operation in the main material number area to delete it. If the personnel determines that the main material number is correct, they do not need to delete the main material number.

[0185] Fig.11 The schematic diagram of the structure of the component assembly abnormality detection device provided in this application includes:

[0186] A first determination module 111 is used to obtain a measured image of the component to be detected, input the measured image into a character region detection model, and determine a first character region sub-image in the measured image based on the character region detection model;

[0187] A second determination module 112 is used to input the first character region sub-image into a character recognition model, and determine first character group information of each row in the first character region sub-image based on the character recognition model;

[0188] The detection module 113 is used to obtain the pre-saved target unique character group information of the target element corresponding to the element to be detected. If the first character group information of each row contains the target unique character group information, it is determined that the element to be detected is assembled normally; otherwise, it is determined that the element to be detected is assembled abnormally.

[0189] The device also includes:

[0190] The third determination module 114 is used to obtain a standard image of the target element, input the standard image into the character area detection model, and determine a second character area sub-image in the standard image based on the character area detection model; input the second character area sub-image into the character recognition model, and determine the second character group information of each row in the second character area sub-image based on the character recognition model; obtain each unique character group information saved in advance; for the second character group information of each row, match the second character group information with each unique character group information, and if the match is successful, determine the second character group information as the target unique character group information of the target element and save it.

[0191] The third determining module 114 is configured to obtain the first text information of each pre-stored unique character group information; and match the second text information of the second character group information with the first text information of each unique character group information.

[0192] The third determination module 114 is used to obtain a pre-stored first regular expression of each unique character group information; and match the second regular expression of the second character group information with the first regular expression of each unique character group information.

[0193] The third determination module 114 is used to obtain the pre-saved non-unique character group information, and for each line of the second character group information, determine whether the second character group information exists in the non-unique character group information, and if so, filter out the second character group information and determine the retained second character group information as the target unique character group information.

[0194] The third determination module 114 is used to obtain the third regular expression of each pre-saved non-unique character group information; for the second character group information of each line, determine whether the second regular expression of the second character group information exists in the third regular expression of the each non-unique character group information, and if so, filter out the second character group information corresponding to the second regular expression, and determine the retained second character group information as the target unique character group information.

[0195] The third determination module 114 is used to input the standard image into the component category detection model, determine the target category information of the target component in the standard image based on the component category detection model; determine the target character length range corresponding to the target category information according to the character length ranges corresponding to each pre-saved category information; determine the character length of the second character group information for each line; determine whether the character length belongs to the target character length range, if not, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

[0196] The third determination module 114 is used for determining the second character group information of a row as the target unique character group information if only a row of second character group information is retained; if multiple rows of second character group information are retained, inputting the second character group information of each row into the character information classification model respectively, and determining the character category information of the second character group information of each row based on the character information classification model; wherein the character category information includes unique information and non-unique information; and determining the second character group information whose character category information is unique information as the target unique character group information.

[0197] The third determination module 114 is used to determine the row number corresponding to the second character group information for each retained row; input the second character group information and the corresponding row number into the character information classification model, and determine the character category information of the second character group information based on the character information classification model; wherein, the corresponding weight parameter is determined according to the row number corresponding to the sample character group information in the training set, and the confidence that the character category information of the sample character group information is the only class information is predicted according to the weight parameter; and the character information classification model is trained according to the confidence and the character category information label corresponding to the sample character group information.

[0198] The detection module 113 is used to determine at least one target replacement unique character group information corresponding to the target unique character group information based on at least one replacement unique character group information corresponding to each pre-saved unique character group information; if the first character group information of each row and the at least one target replacement unique character group information do not have the same character group information, it is determined that the assembly of the component to be detected is abnormal; if the first character group information of each row and the at least one target replacement unique character group information do have the same character group information, it is determined that the assembly of the component to be detected is normal.

[0199] The present application also provides an electronic device, such as Fig.12 As shown, it includes: a processor 121, a communication interface 122, a memory 123 and a communication bus 124, wherein the processor 121, the communication interface 122, and the memory 123 communicate with each other through the communication bus 124;

[0200] The memory 123 stores a computer program, and when the program is executed by the processor 121, the processor 121 executes any one of the above method steps.

[0201] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0202] The communication interface 122 is used for communication between the electronic device and other devices.

[0203] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0204] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0205] The present application also provides a computer storage readable storage medium, wherein the computer readable storage medium stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0206] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0207] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A component assembly anomaly detection method, characterized in that: The method comprises: Acquire a measured image of the component to be detected, input the measured image into a character region detection model, and determine a first character region sub-image in the measured image based on the character region detection model; Inputting the first character region sub-image into a character recognition model, and determining first character group information of each row in the first character region sub-image based on the character recognition model; The pre-saved target unique character group information of the target element corresponding to the element to be detected is obtained. If the first character group information of each row contains the target unique character group information, it is determined that the element to be detected is assembled normally; otherwise, it is determined that the element to be detected is assembled abnormally.

2. The method according to claim 1, characterized in that The process of pre-saving the target unique character group information of the target element includes: Acquire a standard image of the target element, input the standard image into the character region detection model, and determine a second character region sub-image in the standard image based on the character region detection model; Inputting the second character region sub-image into the character recognition model, and determining second character group information of each row in the second character region sub-image based on the character recognition model; Acquire each pre-saved unique character group information; for the second character group information of each row, match the second character group information with each unique character group information, and if the match is successful, determine the second character group information as the target unique character group information of the target element and save it.

3. The method according to claim 2, characterized in that The obtaining of each pre-stored unique character group information includes: Acquire first text information of each unique character group information stored in advance; Matching the second character group information with each of the unique character group information comprises: The second text information of the second character group information is matched with the first text information of each unique character group information.

4. The method according to claim 2 or 3, characterized in that The obtaining of each pre-stored unique character group information includes: Obtaining a first regular expression of each unique character group information saved in advance; Matching the second character group information with each of the unique character group information comprises: The second regular expression of the second character group information is matched with the first regular expression of each unique character group information.

5. The method according to claim 2, characterized in that If there is no successfully matched second character group information, the method further includes: Obtain each pre-saved non-unique character group information, and for each line of second character group information, determine whether the second character group information exists in the each non-unique character group information; if so, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

6. The method according to claim 2, characterized in that If there is no successfully matched second character group information, the method further includes: A third regular expression for obtaining pre-saved information of each non-unique character group; For the second character group information of each row, determine whether the second regular expression of the second character group information exists in the third regular expression of each non-unique character group information; if so, filter out the second character group information corresponding to the second regular expression, and determine the retained second character group information as the target unique character group information.

7. The method according to claim 2, characterized in that If there is no successfully matched second character group information, the method further includes: Inputting the standard image into a component category detection model, and determining target category information of the target component in the standard image based on the component category detection model; Determine the target character length range corresponding to the target category information according to the character length ranges corresponding to the respective category information stored in advance; For the second character group information of each row, determine the character length of the second character group information; judge whether the character length belongs to the target character length range, if not, filter out the second character group information, and determine the retained second character group information as the target unique character group information.

8. The method according to any one of claims 5, 6 and 7, characterized in that The step of determining the retained second character group information as the target unique character group information comprises: If only one row of second character group information is retained, determining the second character group information of the row as the target unique character group information; If multiple lines of second character group information are retained, the method further includes: Inputting the second character group information of each retained line into a character information classification model respectively, and determining character category information of the second character group information of each line based on the character information classification model; wherein the character category information includes unique category information and non-unique category information; The second character group information whose character category information is unique type information is determined as the target unique character group information.

9. The method according to claim 8, characterized in that The step of inputting the retained second character group information of each line into a character information classification model respectively, and determining the character category information of the second character group information of each line based on the character information classification model comprises: For each retained line of second character group information, determining a line number corresponding to the second character group information; inputting the second character group information and the corresponding line number into the character information classification model, and determining character category information of the second character group information based on the character information classification model; Among them, the corresponding weight parameter is determined according to the row number corresponding to the sample character group information in the training set, and the confidence that the character category information of the sample character group information is the only category information is predicted according to the weight parameter; and the character information classification model is trained according to the confidence and the character category information label corresponding to the sample character group information.

10. The method according to claim 1, characterized in that Before determining that the component to be detected is assembled abnormally, the method further includes: Determine at least one target replacement unique character group information corresponding to the target unique character group information according to at least one replacement unique character group information corresponding to each pre-stored unique character group information; If the first character group information of each row and the at least one target replacement unique character group information do not have the same character group information, it is determined that the component to be detected is assembled abnormally; The method further comprises: If the first character group information of each row and the at least one target replacement unique character group information contain the same character group information, it is determined that the component to be inspected is assembled normally.

Images