Part information determination method and device, equipment and medium
By using the center of mass and surrounding rectangle matching technology of the image area in the determination of component information, the problems of high complexity and time-consuming in the prior art are solved, and the rapid and accurate matching of component information is achieved, and efficiency and accuracy are improved.
Patent Information
- Application Number
- CN202411427219.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-14
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, the component information determination solution is complex, time-consuming and difficult to guarantee.
By determining the center of mass, number number and component information of the component image area in the product image template, and storing it in the preset information table; obtaining the pending product image, determining the center of mass and the minimum surrounding rectangle of its image area; according to Production line scene type, determine the image difference type, and match the product images and templates to be processed to obtain target component information.
It realizes fast and accurate matching of component information, reduces complexity and time-consuming, and improves the efficiency and accuracy of the information determination process.
Smart Images

Figure CN118941566A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image data processing, and in particular to a method, device, equipment and medium for determining component information. Background Art
[0002] A product is usually provided with multiple parts. The parts set at each position of the product are fixed, and each part has its fixed part information. Part information may include type information, identification information, defect specifications and other information. During the manufacturing process of the product, the product image needs to be processed to determine the part information of each part in the product image. The part information of each part in the product image can be the part information that the part correctly located at the product position where each part is located should have. By detecting each part in the product image based on the part information of each part in the product image, it can be determined whether there are defective parts or parts of the wrong category in the product, and whether there are parts falling off in the product.
[0003] In the related art, the commonly used solution for determining component information is: processing the product image based on affine transformation technology or image cropping technology to determine the component information of each component in the product image, which is highly complex, time-consuming, and difficult to guarantee accuracy. Summary of the invention
[0004] The present invention provides a method, device, equipment and medium for determining component information to solve the problems in the related art that the component information determination scheme is highly complex, time-consuming and difficult to ensure accuracy.
[0005] According to one aspect of the present invention, there is provided a method for determining component information, comprising:
[0006] Determine the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table accordingly;
[0007] Acquire an image of a product to be processed, and determine an image area of each component in the image of the product to be processed;
[0008] Determine the centroid of the image area of each component in the image of the product to be processed, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle;
[0009] Determining, according to the production line scene type of the product image to be processed, the type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template;
[0010] The product image to be processed and the product image template are matched according to the image difference type, target components in the product image template that match each component in the product image to be processed are determined, and component information of the target components that match each component is obtained from the preset information table as the component information of each component in the product image to be processed.
[0011] According to another aspect of the present invention, there is provided a device for determining component information, comprising:
[0012] A template processing module, used to determine the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table;
[0013] An image acquisition module, used to acquire an image of a product to be processed and determine an image area of each component in the image of the product to be processed;
[0014] A centroid determination module, used to determine the centroid of the image area of each component in the product image to be processed, the centroid corresponding to the centroid of the image area of all components, and a minimum enclosing rectangle;
[0015] A difference type determination module, used to determine the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template according to the production line scene type of the product image to be processed;
[0016] A component information determination module is used to match the product image to be processed and the product image template according to the image difference type, determine the target components in the product image template that match the various components in the product image to be processed, and obtain the component information of the target components that match the various components from the preset information table as the component information of the various components in the product image to be processed.
[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0018] at least one processor;
[0019] and a memory communicatively coupled to the at least one processor;
[0020] The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the component information determination method described in any embodiment of the present invention.
[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the component information determination method described in any embodiment of the present invention when executed.
[0022] The technical solution of the embodiment of the present invention determines the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and stores the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table; after obtaining the product image to be processed, determines the image area of each component in the product image to be processed, determines the centroid of the image area of each component in the product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle; then, according to the production line scene type of the product image to be processed, determines the image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template Type, match the product image to be processed and the product image template according to the image difference type, determine the target parts in the product image template that match the various parts in the product image to be processed, obtain the part information of the target parts matching each part from the preset information table, as the part information of each part in the product image to be processed, solve the problems of high complexity, long time consumption and difficult accuracy of the part information determination scheme in the related technology, and can quickly and accurately match the parts in the product image and the product image template based on the centroid of the image area of each part in the product image and the product image template, and determine the part information of each part in the product image, with low complexity and short time consumption, thereby improving the efficiency and accuracy of the part information determination process.
[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0025] Figure 1 A flowchart of a method for determining component information provided in Embodiment 1 of the present invention.
[0026] Figure 2 A schematic diagram of an image area of a component in a product image template provided in Embodiment 1 of the present invention.
[0027] Figure 3 A schematic diagram of the centroid of an image area of a component in a product image template provided in the first embodiment of the present invention, the centroid corresponding to the centroid of image areas of all components, and a minimum enclosing rectangle.
[0028] Figure 4 A schematic diagram of a preset information table provided in Embodiment 1 of the present invention.
[0029] Figure 5 A schematic diagram of an image region of a component in a product image to be processed provided in Embodiment 1 of the present invention.
[0030] Figure 6 A schematic diagram of a centroid of an image region of a component in a product image to be processed, a centroid corresponding to the centroid of image regions of all components, and a minimum enclosing rectangle provided in a first embodiment of the present invention.
[0031] Figure 7 A schematic diagram of an image region of a component in a product image to be processed provided in Embodiment 1 of the present invention.
[0032] Figure 8 A schematic diagram of a centroid of an image region of a component in a product image to be processed, a centroid corresponding to the centroid of image regions of all components, and a minimum enclosing rectangle provided in a first embodiment of the present invention.
[0033] Fig. 9 A schematic diagram of the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template and the horizontal direction provided in Example 1 of the present invention.
[0034] Fig.10 A schematic diagram of an image region of a component in a product image to be processed provided in Embodiment 1 of the present invention.
[0035] Fig.11 A schematic diagram of a centroid of an image region of a component in a product image to be processed, a centroid corresponding to the centroid of image regions of all components, and a minimum enclosing rectangle provided in a first embodiment of the present invention.
[0036] Fig.12 A schematic diagram of a minimum enclosing rectangle corresponding to the centroid of the image areas of all components in an image area combination provided in Embodiment 1 of the present invention.
[0037] Fig.13 A schematic diagram of a minimum enclosing rectangle corresponding to the centroid of the image areas of all components in an image area combination provided in Embodiment 1 of the present invention.
[0038] Fig.14 A flowchart of a method for determining component information provided in Embodiment 2 of the present invention.
[0039] Fig.15 A schematic diagram of the structure of a device for determining component information provided in Embodiment 3 of the present invention.
[0040] Fig.16 A schematic diagram of the structure of an electronic device for implementing the method for determining component information according to an embodiment of the present invention. DETAILED DESCRIPTION
[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0042] It should be noted that the terms "target", "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprise", "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0043] Embodiment 1
[0044] Figure 1 A flowchart of a component information determination method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case where a product image is processed to determine the component information of each component in the product image. The method can be executed by a component information determination device, which can be implemented in the form of hardware and / or software, and the component information determination device can be configured in an electronic device. The electronic device is used to process the product image during the manufacturing process of the product and determine the component information of each component in the product image. Figure 1 As shown, the method includes:
[0045] Step 101, determine the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table accordingly.
[0046] Optionally, the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component are determined, and the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component are correspondingly stored in a preset information table, including: obtaining the product image template, and determining the image area of each component in the product image template; determining the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle; determining the digital number and component information of each component in the product image template; and storing the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in the preset information table.
[0047] Optionally, the product image is an image obtained by photographing the product and including various components provided in the product. The product may be a hardware device for providing a specified function.
[0048] Optionally, the product image template may be a well-imaged image of each component provided in the qualified product obtained by photographing the qualified product. A qualified product may refer to a product that is not provided with defective components or components of the wrong category, and has no components falling off. Acquiring the product image template includes: acquiring the product image template sent by the target user. The target user may be a technician responsible for managing the product. The target user may send the product image template to the electronic device through the terminal device.
[0049] Optionally, determining the image area of each component in the product image template includes: determining the image area of each component in the product image template through a pre-trained image segmentation model. For each component in the product image template, the image area of the component may refer to the image area in the product image template containing the component. The electronic device is provided with a pre-trained image segmentation model. The pre-trained image segmentation model may be a deep learning model for detecting the product image, segmenting the image area of each component in the product image from the product image, and then outputting the image area of each component in the product image. The input of the pre-trained image segmentation model is the product image, and the output of the pre-trained image segmentation model is the image area of each component in the product image. The electronic device may input the product image template into the pre-trained image segmentation model. The pre-trained image segmentation model detects the product image template, segments the image area of each component in the product image template from the product image template, and then outputs the image area of each component in the product image template. The electronic device obtains the image area of each component in the product image template output by the image segmentation model. The pre-trained image segmentation model may be set in the electronic device by a technician.
[0050] Optionally, in a specific example, Figure 2 A schematic diagram of the image area of a component in a product image template provided in the first embodiment of the present invention. A qualified product is provided with 7 components. The product image template includes 7 component image areas: A1, A2, A3, A4, A5, A6, and A7.
[0051] Optionally, for each component, the centroid of the image area of the component includes a horizontal coordinate and a vertical coordinate. The horizontal coordinate of the centroid of the image area of the component is the average value of the horizontal coordinates in the pixel coordinates of all pixels in the image area of the component. The pixel coordinates of a pixel are the pixel coordinates of the pixel in the product image template. The pixel coordinates include a horizontal coordinate and a vertical coordinate. The horizontal coordinate in the pixel coordinates of a pixel is the number of columns of the pixel in the product image template. The vertical coordinate in the pixel coordinates of a pixel is the number of rows of the pixel in the product image template.
[0052] Optionally, determining the centroid of the image region of the component includes: calculating the centroid of the image region of the component using the following calculation formula:
[0053] ,
[0054] ,
[0055] Among them, the centroid of the image area of the component is , is the abscissa of the centroid of the image area of the component, is the ordinate of the centroid of the image area of the component. The image area of the component contains n pixels, where n is an integer greater than 0. is the horizontal coordinate of the pixel coordinates of the ith pixel point in the image area of the component, is the ordinate of the pixel coordinates of the ith pixel in the image area of the component, .
[0056] Optionally, the centroid corresponding to the centroid of the image area of all parts includes a horizontal coordinate and a vertical coordinate. The horizontal coordinate of the centroid corresponding to the centroid of the image area of all parts is the average value of the horizontal coordinates in the centroid of the image area of all parts. The vertical coordinate of the centroid corresponding to the centroid of the image area of all parts is the average value of the vertical coordinates in the centroid of the image area of all parts.
[0057] Optionally, the minimum enclosing rectangle corresponding to the centroid of the image areas of all parts is the minimum rectangular image area in the product image template that contains the centroid of the image areas of all parts.
[0058] Optionally, a calculation module is provided in the electronic device. The calculation module may be a software module for calculating according to the image area of each component in the product image, determining the centroid of the image area of each component in the product image, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle, and then outputting the centroid of the image area of each component in the product image, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle. The calculation module may be a software module provided by a technician in the electronic device. The electronic device may input the image area of each component in the product image template into the calculation module. The calculation module calculates according to the image area of each component in the product image template, determines the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle, and then outputs the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle. The electronic device obtains the centroid of the image area of each component in the product image template output by the calculation module, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle.
[0059] Optionally, in a specific example, Figure 3A schematic diagram of the centroid of the image area of a component in a product image template provided in Embodiment 1 of the present invention, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle. Points PA1, PA2, PA3, PA4, PA5, PA6, and PA7 are points used to represent the centroid of the image area of the component in the product image template. Point PA11 is a point used to represent the centroid corresponding to the centroid of the image areas of all components. The minimum enclosing rectangle corresponding to the centroid of the image areas of all components is the smallest rectangular image area in the product image template that contains the centroid of the image areas of all components.
[0060] Optionally, determining the digital number and component information of each component in the product image template includes: obtaining the digital number and component information of each component in the product image template sent by the target user. The digital number of each component in the product image template may be a digital number assigned to the component by the target user. The digital numbers of different components are different. The component information of each component in the product image template is the component information that the component correctly located at the product position where the component is located should have. The component information may include type information, identification information, defect specifications and other information. The type information may be a character string used to characterize the type of the component. The identification information may be a character string used to characterize the component. The defect specification may be the area range of the defect area in the component.
[0061] Optionally, for each component in the product image template, the centroid of the component image area, the digital number of the component, and the component information of the component are correspondingly stored in a preset information table. The preset information table may be a table set in the electronic device for storing relevant information of the components in the product image template.
[0062] Optionally, in a specific example, Figure 4 A schematic diagram of a preset information table provided in Embodiment 1 of the present invention. The product image template includes image areas of seven components. For each component in the product image template, the centroid of the image area of the component, the digital number of the component, the type information of the component, the identification information of the component, and the defect specifications of the component are correspondingly stored in the preset information table.
[0063] Step 102: Acquire the image of the product to be processed, and determine the image area of each component in the image of the product to be processed.
[0064] Optionally, the product image to be processed may be a product image that needs to be processed at the current moment.
[0065] Optionally, obtaining the image of the product to be processed includes: receiving the image of the product to be processed sent by a photographing device. The photographing device may be a camera for photographing the product to obtain the image of the product to be processed. The photographing device will send the photographed image of the product to be processed to the electronic device.
[0066] Optionally, obtaining the image of the product to be processed includes: receiving the image of the product to be processed sent by a target user. The target user may be a technician responsible for managing the product. The target user photographs the product through a terminal device to obtain the image of the product to be processed, and then sends the photographed image of the product to be processed to an electronic device.
[0067] Optionally, determining the image areas of each component in the image of the product to be processed includes: determining the image areas of each component in the image of the product to be processed by a pre-trained image segmentation model. For each component in the image of the product to be processed, the image area of the component may refer to the image area of the product image to be processed that contains the component. The electronic device may input the image of the product to be processed into a pre-trained image segmentation model. The pre-trained image segmentation model detects the image of the product to be processed, segments the image areas of each component in the image of the product to be processed from the image of the product to be processed, and then outputs the image areas of each component in the image of the product to be processed. The electronic device obtains the image areas of each component in the image of the product to be processed output by the image segmentation model.
[0068] Step 103: Determine the centroid of the image area of each component in the to-be-processed product image, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle.
[0069] Optionally, the electronic device can input the image area of each component in the image of the product to be processed into the calculation module. The calculation module will perform calculations based on the image area of each component in the image of the product to be processed, determine the centroid of the image area of each component in the image of the product to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle, and then output the centroid of the image area of each component in the image of the product to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle. The electronic device obtains the centroid of the image area of each component in the image of the product to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle output by the calculation module.
[0070] Step 104: Determine the type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template according to the production line scene type of the product image to be processed.
[0071] Optionally, ideally, the image area of the parts in the product image to be processed can overlap with the image area of the parts in the product image template. However, abnormal parts of the product during shooting and / or the placement of the product will cause the image area of the parts in the product image to be processed to be different from the image area of the parts in the product image template, so that the image area of the parts in the product image to be processed cannot overlap with the image area of the parts in the product image template. Among them, the abnormal parts include quantitative abnormalities such as parts falling off or parts mixing. The existence of parts falling off in the product may refer to parts falling off in the product. The existence of parts mixing in the product may refer to the wrong category of parts being set in the product.
[0072] Optionally, the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template may be information used to characterize the difference between the image area of the component in the product image to be processed and the image area of the component in the product image template. The image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template may be translation, translation and rotation, or component anomaly, translation, and rotation.
[0073] Optionally, the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template is translation, indicating that there is a certain translation in the position of the image area of the component in the product image to be processed, so that the image area of the component in the product image to be processed and the image area of the component in the product image template cannot overlap.
[0074] Optionally, the type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template is translation and rotation, indicating that the position of the image area of the component in the product image to be processed has a certain translation and rotation, so that the image area of the component in the product image to be processed and the image area of the component in the product image template cannot overlap.
[0075] Optionally, the image difference types between the image area of the parts in the product image to be processed and the image area of the parts in the product image template are part abnormality, translation and rotation, indicating that the product in the product image to be processed has parts falling off or parts are mixed, and the position of the image area of the parts in the product image to be processed has a certain translation and rotation, so that the image area of the parts in the product image to be processed cannot overlap with the image area of the parts in the product image template.
[0076] Optionally, according to the production line scene type of the product image to be processed, the type of image difference between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is determined, including: if the production line scene type of the product image to be processed is that the product is intact and neatly arranged, then the type of image difference between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is determined to be translation; if the production line scene type of the product image to be processed is that the product is not neatly arranged, then the type of image difference between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is determined to be translation and rotation; if the production line scene type of the product image to be processed is that the product has abnormal parts and is not neatly arranged, then the type of image difference between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is determined to be abnormal parts, translation and rotation.
[0077] Optionally, the production line scene type of the product image to be processed may be sent to the electronic device together with the product image to be processed. The production line scene type of the product image to be processed may be information used to describe the abnormal parts of the product at the time of shooting and / or the placement of the product. The production line scene type of the product image to be processed is that the product is intact and neatly placed, the product is not neatly placed, or the product has abnormal parts and is not neatly placed. Generally, if the production line scene type of the product image to be processed is that the product is intact and neatly placed, it can be determined that the image difference type between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is the presence of translation. If the production line scene type of the product image to be processed is that the product is not neatly placed, it can be determined that the image difference type between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is the presence of translation and rotation. If the production line scene type of the product image to be processed is that the product has abnormal parts and is not neatly placed, it can be determined that the image difference type between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is the presence of parts abnormality, translation and rotation.
[0078] Step 105: Match the product image to be processed and the product image template according to the image difference type, determine the target components in the product image template that match the respective components in the product image to be processed, and obtain component information of the target components that match the respective components from the preset information table as the component information of the respective components in the product image to be processed.
[0079] Optionally, for each component in the product image to be processed, the target component in the product image template that matches the component is the component in the product image template that is located at the same product position as the component. The component information of the target component in the product image template that matches the component is the component information that the component that is correctly located at the product position where the component is located should have.
[0080] Optionally, the product image to be processed and the product image template are matched according to the image difference type, target components in the product image template that match each component in the product image to be processed are determined, and component information of the target components that match each component is obtained from the preset information table as the component information of each component in the product image to be processed, including: if the image difference type is that there is translation, then according to a preset distance calculation formula, the distance between the centroid of the image area of each component in the product image to be processed and the centroid of the image area of each component in the product image template is calculated; for each component in the product image to be processed, the component in the product image template with the smallest value of the distance to the centroid of the image area of the component is determined as the target component in the product image template that matches the component, and the component information of the target component in the product image template that matches the component is obtained from the preset information table as the component information of the component.
[0081] Optionally, calculating the distance between the centroid of the image area of each component in the product image to be processed and the centroid of the image area of each component in the product image template according to a preset distance calculation formula includes: performing the following operations for each component in the product image to be processed: reading the centroid of the image area of each component in the product image template from a preset information table, and calculating the distance between the centroid of the image area of the component and the centroid of the image area of each component in the product image template according to the preset distance calculation formula. The preset distance calculation formula may be a preset formula for calculating the distance between the centroid of the image area of the component in the product image to be processed and the centroid of the image area of the component in the product image template.
[0082] Optionally, calculating the distances between the centroid of the image area of the component and the centroid of the image area of each component in the product image template according to a preset distance calculation formula includes: calculating the distance between the centroid of the image area of the i-th component in the product image to be processed and the centroid of the image area of the j-th component in the product image template according to the following formula:
[0083] ,
[0084] in, is the distance between the centroid of the image area of the i-th component in the product image to be processed and the centroid of the image area of the j-th component in the product image template. The centroid of the image area of the i-th component in the product image to be processed is , is the horizontal coordinate of the centroid of the image area of the i-th component in the product image to be processed, is the ordinate of the centroid of the image area of the i-th component in the product image to be processed, and the centroid of the image area of the j-th component in the product image template is , is the abscissa of the centroid of the image area of the jth component in the product image template, is the ordinate of the centroid of the image area of the jth component in the product image template.
[0085] Optionally, the product image template includes image regions of m components. The product image to be processed includes image regions of m components. . m is an integer greater than 0. For the i-th component in the product image to be processed, the distance between the centroid of the image area of the i-th component and the centroid of the image area of each component in the product image template is calculated respectively: , , … . is the distance between the centroid of the image area of the i-th component in the product image to be processed and the centroid of the image area of the first component in the product image template. It is the distance between the centroid of the image area of the i-th component in the product image to be processed and the centroid of the image area of the second component in the product image template. is the distance between the centroid of the image area of the i-th component in the product image to be processed and the centroid of the image area of the third component in the product image template. And so on, is the distance between the centroid of the image area of the i-th component in the product image to be processed and the centroid of the image area of the m-th component in the product image template. Determine the distance with the smallest value among the calculated distances between the centroid of the image area of the i-th component and the centroid of the image areas of each component in the product image template. . The distance from the centroid of the image area of the i-th component is The component in the product image template is determined as a target component that matches the component in the product image template.
[0086] Optionally, obtaining component information of a target component that matches a component in the product image template from the preset information table as component information of the component, including: performing the following operations for each component in the product image to be processed: reading component information of a target component that matches a component from the preset information table as component information of the component.
[0087] Optionally, in a specific example, Figure 5 A schematic diagram of an image region of a component in a product image to be processed provided in Embodiment 1 of the present invention. The image difference type between the image region of the component in the product image to be processed and the image region of the component in the product image template is translation. The product image to be processed contains image regions of 7 components: B1, B2, B3, B4, B5, B6, and B7.
[0088] Optionally, in a specific example, Figure 6 A schematic diagram of the centroid of the image area of a component in a product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle provided in Embodiment 1 of the present invention. The type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template is translation. Point PB1, point PB2, point PB3, point PB4, point PB5, point PB6, and point PB7 are points used to represent the centroid of the image area of the component in the product image to be processed. Point PB11 is a point used to represent the centroid corresponding to the centroid of the image area of all components. The minimum enclosing rectangle corresponding to the centroid of the image area of all components is the smallest rectangular image area containing the centroid of the image area of all components in the product image to be processed.
[0089] Optionally, the product image to be processed and the product image template are matched according to the image difference type, target parts in the product image template that match each part in the product image to be processed are determined, and part information of the target parts that match each part is obtained from the preset information table as the part information of each part in the product image to be processed, including: if the image difference type is that there is translation and rotation, then according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template and the horizontal direction, the image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed is rotated at multiple angles to obtain the image area of each part in the product image to be processed after rotation at each rotation angle. centroid of the image area of each component in the product image to be processed after rotation; for each rotation angle, according to a preset distance calculation formula, calculate the distance between the centroid of the image area of each component in the product image to be processed after rotation and the centroid of the image area of each component in the product image template, determine the minimum distance corresponding to each component in the product image to be processed, sum the determined minimum distances to obtain a summation result, and determine the minimum summation result among the summation results; determine the rotation angle corresponding to the minimum summation result as the target angle; for each component in the product image to be processed, determine the component in the product image template with the minimum distance from the centroid of the image area of the component after rotation at the target angle as the target component in the product image template that matches the component, and obtain the component information of the target component in the product image template that matches the component from the preset information table as the component information of the component.
[0090] Optionally, according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, the image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template is rotated at multiple angles to obtain the centroid of the image area of each part in the product image to be processed after rotation at each rotation angle, including: determining the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction through an angle detection module. , the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template and the horizontal direction ; Calculate the angle Angle The difference between them gives the angle , ; Through the image rotation module, the image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed is rotated clockwise ,get The image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all the parts in the image of the product to be processed is rotated clockwise by the image rotation module. ,get The image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all the parts in the image of the product to be processed is rotated clockwise by the image rotation module. ,get The image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all the parts in the image of the product to be processed is rotated clockwise by the image rotation module. ,get The centroid of the image area of each component in the rotated product image to be processed.
[0091] Optionally, the angle detection module may be a software module provided in the electronic device for detecting the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image and the horizontal direction. The electronic device may call the angle detection module to determine the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction. , the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template and the horizontal direction .
[0092] Optionally, the image rotation module may be a software module provided in the electronic device for rotating the image area within the minimum enclosing rectangle corresponding to the centroid of the image areas of all parts in the product image by a specified angle clockwise to obtain the centroid of the image areas of each part in the product image after rotation at the specified angle. The electronic device may call the image rotation module to rotate the image area within the minimum enclosing rectangle corresponding to the centroid of the image areas of all parts in the product image to be processed by a specified angle clockwise to obtain the centroid of the image areas of each part in the product image to be processed after rotation at the specified angle.
[0093] Optionally, for each rotation angle, according to a preset distance calculation formula, the distance between the centroid of the image area of each component in the image of the product to be processed after rotation at the rotation angle and the centroid of the image area of each component in the product image template is calculated, and the minimum distance corresponding to the centroid of the image area of each component in the image of the product to be processed after rotation at the rotation angle is determined, and the determined minimum distances are summed to obtain a summation result:
[0094] ,
[0095] in, For the determined The summation result is obtained by summing up the minimum distances corresponding to the centroids of the image areas of each component in the rotated product image to be processed. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the first component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the second component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the third component in the product image template. And so on, for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the m-th component in the product image template. yes The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of each component in the product image template is the smallest distance, that is, The minimum distance corresponding to the centroid of the image area of the i-th component in the rotated product image to be processed. For the determined The summation result is obtained by summing up the minimum distances corresponding to the centroids of the image areas of each component in the rotated product image to be processed. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the first component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the second component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the third component in the product image template. And so on, for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the m-th component in the product image template. yes The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of each component in the product image template is the smallest distance, that is, The minimum distance corresponding to the centroid of the image area of the i-th component in the rotated product image to be processed. For the determined The summation result is obtained by summing up the minimum distances corresponding to the centroids of the image areas of each component in the rotated product image to be processed. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the first component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the second component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the third component in the product image template. And so on, for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the m-th component in the product image template. yes The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of each component in the product image template is the smallest distance, that is, The minimum distance corresponding to the centroid of the image area of the i-th component in the rotated product image to be processed. For the determined The summation result is obtained by summing up the minimum distances corresponding to the centroids of the image areas of each component in the rotated product image to be processed. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the first component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the second component in the product image template. for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the third component in the product image template. And so on, for The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of the m-th component in the product image template. yes The distance between the centroid of the image area of the i-th component in the rotated product image to be processed and the centroid of the image area of each component in the product image template is the smallest distance, that is, The minimum distance corresponding to the centroid of the image area of the i-th component in the rotated product image to be processed under the condition. The product image template contains the image area of m components. The product image to be processed contains the image area of m components.
[0096] The summation result with the smallest value among the summation results is determined, and the rotation angle corresponding to the summation result with the smallest value is determined as the target angle. The rotation angle with the smallest summation result is the rotation angle corresponding to the summation result with the smallest value. for , , as well as The smallest sum result in . . The rotation angle with the smallest sum is determined as the target angle. For each component in the product image to be processed, the component in the product image template with the smallest value of the distance to the centroid of the image area of the component after rotation at the target angle is determined as the target component that matches the component in the product image template, and the component information of the target component that matches the component in the product image template is obtained from the preset information table as the component information of the component.
[0097] Optionally, in a specific example, Figure 7A schematic diagram of an image region of a component in a product image to be processed provided in Embodiment 1 of the present invention. The image difference type between the image region of the component in the product image to be processed and the image region of the component in the product image template is translation and rotation. The product image to be processed contains image regions of 7 components: C1, C2, C3, C4, C5, C6, and C7.
[0098] Optionally, in a specific example, Figure 8 A schematic diagram of the centroid of the image area of a component in a product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle provided in the first embodiment of the present invention. The type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template is translation and rotation. Point PC1, point PC2, point PC3, point PC4, point PC5, point PC6, and point PC7 are points used to represent the centroid of the image area of the component in the product image to be processed. Point PC11 is a point used to represent the centroid corresponding to the centroid of the image area of all components. The minimum enclosing rectangle corresponding to the centroid of the image area of all components is the smallest rectangular image area containing the centroid of the image area of all components in the product image to be processed.
[0099] Optionally, in a specific example, Fig. 9 A schematic diagram of the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template and the horizontal direction provided in Example 1 of the present invention. It is the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction. It is the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template and the horizontal direction. is a point representing the centroid corresponding to the centroid of the image area of all parts in the product image to be processed. Point is a point representing the centroid corresponding to the centroid of the image areas of all parts in the product image template.
[0100] Optionally, the product image to be processed and the product image template are matched according to the image difference type, target components in the product image template that match each component in the product image to be processed are determined, and component information of the target components that match each component is obtained from the preset information table as the component information of each component in the product image to be processed, including: if the image difference type is component anomaly, translation, and rotation, then according to the number of image areas of the components in the product image to be processed, the image areas of the components are selected from the image areas of all components in the product image template for combination to obtain a plurality of image area combinations; for each image area combination, the centroid of the image area of each component in the image area combination, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle are determined; for each image area combination, according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all components in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all components in the image area combination and the horizontal direction, the minimum enclosing rectangle corresponding to the centroid of the image areas of all components in the product image to be processed is selected and combined. The image area within the enclosing rectangle is rotated at multiple angles to obtain the centroid of the image area of each component in the product image to be processed after rotation at each rotation angle, and for each rotation angle, the distance between the centroid of the image area of each component in the product image to be processed after rotation and the centroid of the image area of each component in the image area combination is calculated according to a preset distance calculation formula, and the minimum distance corresponding to each component in the product image to be processed is determined, and the minimum distance is summed to obtain a sum result, and the minimum sum result is determined in each sum result; the image area combination with the minimum sum result corresponding to the minimum value is determined as the target image area combination; the rotation angle corresponding to the minimum sum result with the target image area combination is determined as the target angle; for each component in the product image to be processed, the component in the target image area combination with the minimum distance between the centroid of the image area of the component after rotation at the target angle is determined as the target component matching the component in the product image template, and the component information of the target component matching the component in the product image template is obtained from the preset information table as the component information of the component.
[0101] Optionally, under normal circumstances, if the image difference type is the presence of component anomalies, translation, and rotation, the number of image areas of components in the product image to be processed is less than the number of image areas of components in the product image template. The number of image areas of components in the product image to be processed is n. The number of image areas of components in the product image template is m. n and m are integers greater than 0. n is less than m. Based on the number of image areas of components in the product image to be processed, image areas of components are selected from the image areas of all components in the product image template and combined to obtain multiple image area combinations, including: selecting n image areas of components from the image areas of all components in the product image template and combining to obtain image region combinations. Each image region combination contains image regions of n components. The image regions of n components contained in each image region combination are not completely the same. .
[0102] Optionally, the following operations are performed for each image area combination: the image areas of each component in the image area group are input into the calculation module, and the centroid of the image area of each component in the image area group output by the calculation module, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle are obtained; the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all components in the image of the product to be processed and the horizontal direction is determined by the angle detection module , the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all components in the image area combination and the horizontal direction ; Calculate the angle Angle The difference between them gives the angle , ; Through the image rotation module, the image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed is rotated clockwise ,get The image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all the parts in the image of the product to be processed is rotated clockwise by the image rotation module. ,get The image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all the parts in the image of the product to be processed is rotated clockwise by the image rotation module. ,get The image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all the parts in the image of the product to be processed is rotated clockwise by the image rotation module. ,get The method comprises the following steps: determining the centroid of the image area of each component in the image of the product to be processed after the rotation at the rotation angle; calculating the distance between the centroid of the image area of each component in the image of the product to be processed after the rotation at the rotation angle and the centroid of the image area of each component in the image area combination according to a preset distance calculation formula for each rotation angle, determining the minimum distance corresponding to the centroid of the image area of each component in the image of the product to be processed after the rotation at the rotation angle, summing up the determined minimum distances to obtain a summation result; and determining the minimum summation result among the summation results.
[0103] Optionally, in a specific example, Fig.10 A schematic diagram of an image region of a component in a product image to be processed provided in Embodiment 1 of the present invention. The image difference types between the image region of the component in the product image to be processed and the image region of the component in the product image template are component anomaly, translation, and rotation. The product image to be processed contains image regions of 6 components: D1, D2, D3, D4, D5, and D6.
[0104] Optionally, in a specific example, Fig.11 A schematic diagram of the centroid of the image area of a component in a product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle provided in the first embodiment of the present invention. The types of image differences between the image area of the component in the product image to be processed and the image area of the component in the product image template are component anomalies, translation, and rotation. Point PD1, point PD2, point PD3, point PD4, point PD5, and point PD6 are points used to represent the centroid of the image area of the component in the product image to be processed. Point PD11 is a point used to represent the centroid corresponding to the centroid of the image area of all components. The minimum enclosing rectangle corresponding to the centroid of the image area of all components is the smallest rectangular image area containing the centroid of the image area of all components in the product image to be processed.
[0105] Optionally, in a specific example, Fig.12 A schematic diagram of a minimum enclosing rectangle corresponding to the centroid of the image areas of all components in an image area combination provided in Embodiment 1 of the present invention.
[0106] Optionally, in a specific example, Fig.13A schematic diagram of a minimum enclosing rectangle corresponding to the centroid of the image areas of all components in an image area combination provided in Embodiment 1 of the present invention.
[0107] Optionally, after determining that the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template is the presence of component anomaly, translation, and rotation, it also includes: sending a preset alarm message to the terminal device of the target user. The preset alarm message can be pre-set information for prompting the presence of detached components or unrecognized components. The terminal device of the target user is a terminal device used by the target user.
[0108] The technical solution of the embodiment of the present invention determines the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and stores the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table; after obtaining the product image to be processed, determines the image area of each component in the product image to be processed, determines the centroid of the image area of each component in the product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle; then, according to the production line scene type of the product image to be processed, determines the image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template Type, match the product image to be processed and the product image template according to the image difference type, determine the target parts in the product image template that match the various parts in the product image to be processed, obtain the part information of the target parts matching each part from the preset information table, as the part information of each part in the product image to be processed, solve the problems of high complexity, long time consumption and difficult accuracy of the part information determination scheme in the related technology, and can quickly and accurately match the parts in the product image and the product image template based on the centroid of the image area of each part in the product image and the product image template, and determine the part information of each part in the product image, with low complexity and short time consumption, thereby improving the efficiency and accuracy of the part information determination process.
[0109] Embodiment 2
[0110] Fig.14 This is a flowchart of a method for determining component information provided by Embodiment 2 of the present invention. This embodiment of the present invention can be combined with each optional solution in one or more of the above embodiments. Fig.14 As shown, the method includes:
[0111] Step 201: Acquire a product image template and determine the image area of each component in the product image template.
[0112] Step 202: Determine the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle.
[0113] Step 203: Determine the digital number and component information of each component in the product image template.
[0114] Step 204: Store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table.
[0115] Step 205: Acquire the image of the product to be processed, and determine the image area of each component in the image of the product to be processed.
[0116] Step 206: Determine the centroid of the image area of each component in the to-be-processed product image, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle.
[0117] Step 207: Determine the type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template according to the production line scene type of the product image to be processed.
[0118] Step 208: Match the product image to be processed and the product image template according to the image difference type, determine the target components in the product image template that match the respective components in the product image to be processed, and obtain component information of the target components that match the respective components from the preset information table as the component information of the respective components in the product image to be processed.
[0119] The technical solution of the embodiment of the present invention can determine the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle, and can store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table. Based on the centroid of the image area of the product image and each component in the product image template, the product image and the components in the product image template can be matched quickly and accurately to determine the component information of each component in the product image, which has low complexity and short time consumption, and improves the efficiency and accuracy of the component information determination process.
[0120] Embodiment 3
[0121] Fig.15 This is a schematic diagram of the structure of a device for determining component information provided by Embodiment 3 of the present invention. The device may be configured in an electronic device. Fig.15 As shown, the device includes: a template processing module 301, an image acquisition module 302, a centroid determination module 303, a difference type determination module 304 and a component information determination module 305.
[0122] The template processing module 301 is used to determine the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in the preset information table; the image acquisition module 302 is used to acquire the product image to be processed and determine the image area of each component in the product image to be processed; the centroid determination module 303 is used to determine the centroid of the image area of each component in the product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum envelope rectangle; a difference type determination module 304, used to determine the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template according to the production line scene type of the product image to be processed; a component information determination module 305, used to match the product image to be processed and the product image template according to the image difference type, determine the target components in the product image template that match each component in the product image to be processed, and obtain the component information of the target components that match each component from the preset information table as the component information of each component in the product image to be processed.
[0123] The technical solution of the embodiment of the present invention determines the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and stores the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table; after obtaining the product image to be processed, determines the image area of each component in the product image to be processed, determines the centroid of the image area of each component in the product image to be processed, the centroid corresponding to the centroid of the image area of all components, and the minimum enclosing rectangle; then, according to the production line scene type of the product image to be processed, determines the image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template Type, match the product image to be processed and the product image template according to the image difference type, determine the target parts in the product image template that match the various parts in the product image to be processed, obtain the part information of the target parts matching each part from the preset information table, as the part information of each part in the product image to be processed, solve the problems of high complexity, long time consumption and difficult accuracy of the part information determination scheme in the related technology, and can quickly and accurately match the parts in the product image and the product image template based on the centroid of the image area of each part in the product image and the product image template, and determine the part information of each part in the product image, with low complexity and short time consumption, thereby improving the efficiency and accuracy of the part information determination process.
[0124] In an optional implementation of an embodiment of the present invention, optionally, the template processing module 301 is specifically used to: obtain a product image template, determine the image area of each component in the product image template; determine the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle; determine the digital number and component information of each component in the product image template; store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table.
[0125] In an optional implementation of an embodiment of the present invention, optionally, the difference type determination module 304 is specifically used to: if the production line scene type of the product image to be processed is that the product is intact and neatly arranged, then determine the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template as translation; if the production line scene type of the product image to be processed is that the product is not neatly arranged, then determine the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template as translation and rotation; if the production line scene type of the product image to be processed is that the product has abnormal components and is not neatly arranged, then determine the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template as component abnormality, translation and rotation.
[0126] In an optional implementation of an embodiment of the present invention, optionally, the component information determination module 305 is specifically used to: if the image difference type is translation, then according to a preset distance calculation formula, calculate the distance between the centroid of the image area of each component in the product image to be processed and the centroid of the image area of each component in the product image template; for each component in the product image to be processed, determine the component in the product image template with the smallest value of the distance between the centroid of the image area of the component as the target component in the product image template that matches the component, and obtain the component information of the target component in the product image template that matches the component from the preset information table as the component information of the component.
[0127] In an optional implementation of the embodiment of the present invention, optionally, the component information determination module 305 is specifically used to: if the image difference type is that there is translation and rotation, then according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all components in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all components in the product image template and the horizontal direction, the image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all components in the product image to be processed is rotated at multiple angles to obtain the centroid of the image area of each component in the product image to be processed after rotation at each rotation angle; for each rotation angle, according to a preset distance calculation formula, calculate the image distance of each component in the product image to be processed after rotation. The method comprises the following steps: determining the distance between the centroid of the image area and the centroid of the image area of each component in the product image template, determining the minimum distance corresponding to each component in the product image to be processed, summing up the determined minimum distances to obtain a summation result, and determining the minimum summation result among the summation results; determining the rotation angle corresponding to the minimum summation result as the target angle; for each component in the product image to be processed, determining the component in the product image template with the minimum distance between the centroid of the image area of the rotated component at the target angle as the target component matching the component in the product image template, and obtaining the component information of the target component matching the component in the product image template from the preset information table as the component information of the component.
[0128] In an optional implementation of the embodiment of the present invention, optionally, the component information determination module 305 is specifically used for: if the image difference type is the presence of component anomaly, translation and rotation, then according to the number of image areas of the components in the image of the product to be processed, the image areas of the components are selected from the image areas of all the components in the product image template for combination to obtain a plurality of image area combinations; for each image area combination, the centroid of the image area of each component in the image area combination, the centroid corresponding to the centroid of the image areas of all the components, and the minimum enclosing rectangle are determined; for each image area combination, according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all the components in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all the components in the image area combination and the horizontal direction, the image area within the minimum enclosing rectangle corresponding to the centroid of the image areas of all the components in the product image to be processed is rotated at multiple angles to obtain each component in the product image to be processed after rotation at each rotation angle. The centroid of the image area is obtained, and for each rotation angle, according to a preset distance calculation formula, the distance between the centroid of the image area of each component in the image to be processed after rotation and the centroid of the image area of each component in the image area combination is calculated, and the minimum distance corresponding to each component in the image to be processed is determined, and the determined minimum distances are summed to obtain a summation result, and the minimum summation result among the summation results is determined; the image area combination with the minimum summation result corresponding to the minimum value is determined as the target image area combination; the rotation angle corresponding to the minimum summation result with the target image area combination is determined as the target angle; for each component in the image to be processed, the component in the target image area combination with the minimum distance between the centroid of the image area of the rotated component at the target angle is determined as the target component matching the component in the product image template, and the component information of the target component matching the component in the product image template is obtained from the preset information table as the component information of the component.
[0129] In an optional implementation of the embodiment of the present invention, optionally, the image acquisition module 302 is specifically used to: determine the image area of each component in the image of the product to be processed by using a pre-trained image segmentation model.
[0130] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0131] The above-mentioned component information determination device can execute the component information determination method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the component information determination method.
[0132] Embodiment 4
[0133] Fig.16 The schematic diagram of the structure of the electronic device 10 that can be used to implement the component information determination method of the embodiment of the present invention is shown. The components, their connections and relationships, and their functions shown herein are only examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0134] like Fig.16 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11 in communication, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program built from the storage unit 18 to the random access memory (RAM) 13. In the random access memory (RAM) 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the read-only memory (ROM) 12, and the random access memory (RAM) 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0135] A number of components in the electronic device 10 are connected to an input / output (I / O) interface 15, including an input unit 16, such as a keyboard, a mouse, etc., an output unit 17, such as various types of displays, speakers, etc., a storage unit 18, such as a disk, an optical disk, etc., and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0136] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a component information determination method.
[0137] In some embodiments, the component information determination method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via a read-only memory (ROM) 12 and / or a communication unit 19. When the computer program is built into a random access memory (RAM) 13 and executed by the processor 11, one or more steps of the component information determination method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the component information determination method in any other appropriate manner (e.g., by means of firmware).
[0138] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0139] The computer program for implementing the component information determination method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed completely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or completely on a remote machine or server.
[0140] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0141] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0142] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0143] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0144] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0145] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for determining component information, characterized in that: include: Determine the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table accordingly; Acquire an image of a product to be processed, and determine an image area of each component in the image of the product to be processed; Determine the centroid of the image area of each component in the image of the product to be processed, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle; Determining, according to the production line scene type of the product image to be processed, the type of image difference between the image area of the component in the product image to be processed and the image area of the component in the product image template; The product image to be processed and the product image template are matched according to the image difference type, target components in the product image template that match each component in the product image to be processed are determined, and component information of the target components that match each component is obtained from the preset information table as the component information of each component in the product image to be processed.
2. The method for determining component information according to claim 1, characterized in that: Determining the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and storing the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in the preset information table accordingly, including: Acquire a product image template, and determine the image area of each component in the product image template; Determine the centroid of the image area of each component in the product image template, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle; Determine the digital number and component information of each component in the product image template; The centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component are correspondingly stored in a preset information table.
3. The method for determining component information according to claim 1, characterized in that: Determining the type of image difference between an image region of a component in the image of the product to be processed and an image region of a component in the product image template according to the production line scene type of the image of the product to be processed includes: If the production line scene type of the product image to be processed is that the products are intact and neatly arranged, determining that the image difference type between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is that there is translation; If the production line scene type of the product image to be processed is that the products are not neatly arranged, determining that the image difference type between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is that there is translation and rotation; If the production line scene type of the product image to be processed is that the product has abnormal parts and is not neatly arranged, then the image difference type between the image area of the parts in the product image to be processed and the image area of the parts in the product image template is determined to be the existence of abnormal parts, translation and rotation.
4. The method for determining component information according to claim 3, characterized in that: Matching the product image to be processed and the product image template according to the image difference type, determining target components in the product image template that match each component in the product image to be processed, and acquiring component information of the target components that match each component from the preset information table as component information of each component in the product image to be processed, including: If the image difference type is translation, the distance between the centroid of the image area of each component in the to-be-processed product image and the centroid of the image area of each component in the product image template is calculated according to a preset distance calculation formula; For each component in the product image to be processed, the component in the product image template having the smallest value of the distance to the centroid of the image area of the component is determined as the target component that matches the component in the product image template, and the component information of the target component that matches the component in the product image template is obtained from the preset information table as the component information of the component.
5. The method for determining component information according to claim 3, characterized in that: Matching the product image to be processed and the product image template according to the image difference type, determining target components in the product image template that match each component in the product image to be processed, and acquiring component information of the target components that match each component from the preset information table as component information of each component in the product image to be processed, including: If the image difference type is that there is translation and rotation, then according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image to be processed and the horizontal direction, the image area within the minimum enclosing rectangle corresponding to the centroid of the image area of all parts in the product image template is rotated at multiple angles to obtain the centroid of the image area of each part in the product image to be processed after rotation at each rotation angle; For each rotation angle, according to a preset distance calculation formula, the distance between the centroid of the image area of each component in the image of the product to be processed after rotation and the centroid of the image area of each component in the product image template is calculated, and the minimum distance corresponding to each component in the image of the product to be processed is determined, and the minimum distances determined are summed to obtain a summation result, and the minimum summation result among the summation results is determined; The rotation angle corresponding to the sum result with the smallest value is determined as the target angle; For each component in the product image to be processed, the component in the product image template having the smallest value of the distance between the component and the centroid of the image area of the component rotated at the target angle is determined as the target component that matches the component in the product image template, and the component information of the target component that matches the component in the product image template is obtained from the preset information table as the component information of the component.
6. The method for determining component information according to claim 3, characterized in that: Matching the product image to be processed and the product image template according to the image difference type, determining target components in the product image template that match each component in the product image to be processed, and acquiring component information of the target components that match each component from the preset information table as component information of each component in the product image to be processed, including: If the image difference type is the presence of component anomaly, translation, and rotation, then according to the number of image areas of components in the product image to be processed, image areas of components are selected from the image areas of all components in the product image template for combination to obtain a plurality of image area combinations; For each image area combination, determining the centroid of the image area of each component in the image area combination, the centroid corresponding to the centroid of the image areas of all components, and the minimum enclosing rectangle; For each image area combination, according to the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all parts in the product image to be processed and the horizontal direction, and the angle between the minimum enclosing rectangle corresponding to the centroid of the image areas of all parts in the image area combination and the horizontal direction, the image area within the minimum enclosing rectangle corresponding to the centroid of the image areas of all parts in the product image to be processed is rotated at multiple angles to obtain the centroid of the image area of each part in the product image to be processed after rotation at each rotation angle, and according to a preset distance calculation formula, the distance between the centroid of the image area of each part in the product image to be processed after rotation and the centroid of the image area of each part in the image area combination is calculated to determine the minimum distance corresponding to each part in the product image to be processed, sum the determined minimum distances to obtain a sum result, and determine the minimum sum result among the sum results; The image region combination with the smallest corresponding value and the smallest sum result is determined as the target image region combination; Determine the rotation angle corresponding to the sum of the minimum value of the combination with the target image area as the target angle; For each component in the product image to be processed, the component in the target image area combination with the smallest value of the distance between the centroid of the image area of the component rotated at the target angle is determined as the target component that matches the component in the product image template, and the component information of the target component that matches the component in the product image template is obtained from the preset information table as the component information of the component.
7. The method for determining component information according to claim 1, characterized in that: Determining the image area of each component in the image of the product to be processed includes: The image regions of the various components in the image of the product to be processed are determined by using a pre-trained image segmentation model.
8. A component information determination device, characterized in that: include: A template processing module, used to determine the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component, and store the centroid of the image area of each component in the product image template, the digital number of each component, and the component information of each component in a preset information table; An image acquisition module, used to acquire an image of a product to be processed and determine an image area of each component in the image of the product to be processed; A centroid determination module, used to determine the centroid of the image area of each component in the product image to be processed, the centroid corresponding to the centroid of the image area of all components, and a minimum enclosing rectangle; A difference type determination module, used to determine the image difference type between the image area of the component in the product image to be processed and the image area of the component in the product image template according to the production line scene type of the product image to be processed; A component information determination module is used to match the product image to be processed and the product image template according to the image difference type, determine the target components in the product image template that match the various components in the product image to be processed, and obtain the component information of the target components that match the various components from the preset information table as the component information of the various components in the product image to be processed.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the component information determination method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the component information determination method according to any one of claims 1 to 7 when executed.
Citation Information
Patent Citations
Industrial part pose recognition method and device, electronic equipment and storage medium
CN113192097A
3D model part number determination method and device, equipment and medium
CN117272414A
Focusing method and apparatus, electronic device, and computer readable storage medium
WO2021057652A1
Image processing method and apparatus, and electronic device
WO2022237657A1
