Flat material installation detection method, system and terminal device

Abstract

The application relates to the technical field of visual identification, and discloses a flat plate material installation detection method, a system and terminal equipment, the flat plate material installation detection method comprising the following steps: inputting a flat plate whole machine image into a target identification model to obtain an identification result, the identification result comprising a flat plate whole machine contour in the flat plate whole machine image; correcting the flat plate whole machine image according to a coordinate conversion matrix to obtain a flat plate whole machine correction image; performing image segmentation on the flat plate whole machine correction image according to a detection template to obtain a plurality of target detection images; and inputting each target detection image into a corresponding material detection model according to a material type to obtain a detection result of corresponding material in the flat plate. The flat plate material installation detection method can complete the detection of multiple materials at the same time after one-time image acquisition, correct position deviation, inclination and angle error caused by manual placement, and reduce the false detection rate and the missed detection rate.

Description

Technical Field

[0001] This application relates to the field of visual recognition technology, and in particular to a method, system and terminal device for detecting the installation of flat panel materials. Background Technology

[0002] The assembly of tablet computers involves installing various materials and components. Due to the wide variety of materials, their dispersed locations, and significant size differences, as well as the small size of some materials or their color blending into the background, issues such as missing, incorrect, or misaligned components after assembly are difficult to identify manually.

[0003] The following problems typically exist in existing production lines: 1. Manual inspection is inefficient. Manually verifying the correct assembly of each component is time-consuming and can easily become a bottleneck under high-paced production conditions. 2. Manual inspection suffers from poor consistency. Differences in experience, attention span, and judgment standards among operators can lead to inconsistent inspection standards for the same material at different times and by different personnel, resulting in missed or false positives. 3. Random placement leads to misjudgments. In actual production lines, tablet computers to be inspected are typically placed in the inspection area by operators, making it difficult to maintain perfect consistency in position, angle, and tilt. Directly identifying a localized area based on fixed coordinates can easily cause target misalignment, leading to inaccurate subsequent inspection results. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, system and terminal device for detecting the installation of flat materials, which can effectively solve the problems of missed detection and false detection in traditional flat material installation detection.

[0005] In a first aspect, embodiments of this application provide a method for detecting the installation of flat panel materials, including: Acquire an image of the assembled tablet, input the image into a target recognition model, and obtain the recognition result output by the target recognition model. The recognition result includes the outline of the tablet in the image. The image of the flat panel is corrected based on the coordinate transformation matrix between the outline of the flat panel and the outline of the standard flat panel to obtain a corrected image of the flat panel. The calibration image of the flatbed machine is segmented according to the detection template of each material of the flatbed machine to obtain multiple target detection images, wherein the target detection image is a region image of each material. Each of the target detection images is input into the corresponding material detection model according to the material type to obtain the detection result of the corresponding material in the plate; The detection template includes a detection confidence threshold for the material at each installation position; the detection result includes one of the following detection types: normal material installation and abnormal material installation; the detection result also includes the model detection confidence for each installation position; and the method further includes: The model detection confidence score for each mounting position is compared with the corresponding detection confidence threshold. If the detection confidence level of the model is greater than or equal to the detection confidence level threshold, the detection type output by the material detection model shall be used as the material detection result of the installation position. If the model detection confidence level is less than the detection confidence threshold, the material detection result at the installation location will be determined as needing re-inspection or installation abnormality. Among them, the detection confidence thresholds for the same type of material at different installation positions are different.

[0006] In a first possible embodiment of the first aspect, acquiring an image of the assembled flat panel includes: Acquire multiple frames of initial flat panel images and image parameters for each frame of the initial flat panel image; Determine whether the image parameters meet the corresponding preset threshold, and if the image parameters meet the corresponding preset threshold multiple times consecutively, obtain the whole image of the tablet. The image parameters include one or more of the following: the offset between the center position of the tablet outline in the initial tablet image and the center position of the standard tablet outline; the outline rotation angle between the tablet outline in the initial tablet image and the standard tablet outline; the area change rate between the area of ​​the tablet outline in the current frame and the area of ​​the tablet outline in the previous frame; and the sharpness of the initial tablet image.

[0007] In a second possible embodiment of the first aspect, before correcting the tablet image based on the coordinate transformation matrix between the tablet's overall profile and the standard tablet's overall profile, the method further includes: The first coordinates of the target pixel in the overall outline of the tablet are obtained, and the second coordinates of the target pixel in the standard overall outline of the tablet are obtained. The target pixel includes one or more of the corner points, boundary points and feature matching points of the overall outline of the tablet and the standard overall outline of the tablet. The coordinate transformation matrix for correcting the whole image of the tablet is calculated based on the first and second coordinates of at least four sets of target pixels.

[0008] In a third possible embodiment of the first aspect, correcting the tablet image based on the coordinate transformation matrix between the tablet's overall outline and the standard tablet's overall outline includes: Based on the coordinate transformation matrix, the coordinates of all pixels in the whole image of the tablet are transformed to obtain the corrected image of the whole tablet.

[0009] In a fourth possible embodiment of the first aspect, the training process of the target recognition model includes: Multiple images of the first tablet are acquired, and a recognition target is labeled in each of the first tablet images to obtain a first dataset. The recognition target includes the outline of the tablet and also includes one or more combinations of the tablet's camera and battery. The target recognition model is obtained by training the model based on the first dataset.

[0010] In a fifth possible embodiment of the first aspect, the detection template includes the cutting dimensions and center coordinates of each material mounting position in the standard flatbed machine image, and the image segmentation of the flatbed machine calibration image based on the detection template of each material of the flatbed machine includes: The target detection image is obtained by segmenting the whole machine calibration image of the flat plate based on the cutting size and center coordinates of each material installation position.

[0011] In a sixth possible embodiment of the first aspect, the training process of the material detection model includes: Acquire a second image of a flatbed machine with normal material installation and a third image of a flatbed machine with abnormal material installation, wherein the abnormal material installation includes missing and / or incorrectly installed target materials. The installation locations of the same material type are marked and cropped in the second and third flat panel whole images to obtain a second dataset. The second dataset includes positive and negative samples for each material type. The positive samples are marked image samples when the target material is installed correctly, and the negative samples are marked image samples when the target material is missing and / or misinstalled. A binary classification model is trained based on the positive and negative samples to obtain a material detection model for the corresponding material type.

[0012] Secondly, embodiments of this application provide a flat panel material installation and detection system, comprising: The target recognition module is used to acquire an image of the assembled tablet, input the image of the assembled tablet into the target recognition model, and obtain the recognition result output by the target recognition model. The recognition result includes the outline of the tablet in the image of the assembled tablet. The image correction module is used to correct the image of the flat panel according to the coordinate transformation matrix between the outline of the flat panel and the outline of the standard flat panel, so as to obtain a corrected image of the flat panel. The image segmentation module is used to segment the corrected image of the flat panel machine according to the detection template of each material of the flat panel machine to obtain multiple target detection images, wherein the target detection image is a region image of each material. The material detection module is used to input each of the target detection images into the corresponding material detection model according to the material type, so as to obtain the detection result of the corresponding material in the plate. The detection template includes a detection confidence threshold for the material at each installation position, the detection result includes one of the detection types: normal material installation and abnormal material installation, and the detection result also includes the model detection confidence for each installation position. The material detection module is further configured to compare the model detection confidence level of each installation position with the corresponding detection confidence level threshold; if the model detection confidence level is greater than or equal to the detection confidence level threshold, the detection type output by the material detection model is used as the material detection result of the installation position; if the model detection confidence level is less than the detection confidence level threshold, the material detection result of the installation position is determined as requiring re-inspection or installation abnormality; wherein, the detection confidence level thresholds corresponding to the same type of material in different installation positions are different.

[0013] In a first possible embodiment of the second aspect, the target recognition module is further configured to acquire multiple frames of initial tablet full-device images and image parameters of each frame of the initial tablet full-device images; determine whether the image parameters meet the corresponding preset threshold; and acquire the tablet full-device image if the image parameters meet the corresponding preset threshold multiple times consecutively. The image parameters include one or more of the following: the offset between the center position of the tablet outline in the initial tablet image and the center position of the standard tablet outline; the outline rotation angle between the tablet outline in the initial tablet image and the standard tablet outline; the area change rate between the area of ​​the tablet outline in the current frame and the area of ​​the tablet outline in the previous frame; and the sharpness of the initial tablet image.

[0014] Thirdly, embodiments of this application provide a terminal device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for detecting the installation of flat materials.

[0015] The embodiments of this application have the following beneficial effects: This embodiment of a method for detecting the installation of flat panel materials includes: acquiring an image of the assembled flat panel; inputting the image into a target recognition model to obtain a recognition result output by the target recognition model, the recognition result including the outline of the flat panel in the image; correcting the image based on the coordinate transformation matrix between the outline and a standard flat panel outline to obtain a corrected image; segmenting the corrected image based on the detection template of each material in the flat panel to obtain multiple target detection images, each target detection image being a region image of a material; and inputting each target detection image into a corresponding material detection model according to the material type to obtain the detection result of the corresponding material in the flat panel.

[0016] Based on the above scheme, this flat panel material installation and inspection method obtains multiple target inspection images through image segmentation, enabling the simultaneous inspection of multiple materials after a single image acquisition. This reduces the time required for item-by-item inspection and meets production line cycle time requirements. By correcting the overall flat panel image, positional offsets, tilts, and angular errors caused by manual placement can be corrected, making the acquisition of subsequent target inspection images more stable. Moreover, this flat panel material installation and inspection method does not rely on directly inspecting all materials on the entire image; instead, it first locates the area image of the material and then performs local inspection. This helps to solve the problems of missed and false detection of small-sized or weak-feature materials such as screws, conductive cloth, and Mylar, reducing the false detection rate and missed detection rate. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This paper illustrates a first flowchart of the plate material installation and testing method according to an embodiment of this application. Figure 2 This paper illustrates a second flowchart of the plate material installation and testing method according to an embodiment of the present application. Figure 3 This illustration shows a first schematic diagram of the installation and testing results of flat panel materials according to an embodiment of this application; Figure 4 This illustrates a second schematic diagram of the flat panel material installation test results according to an embodiment of this application; Figure 5 A schematic diagram of a flat material installation and detection system according to an embodiment of this application is shown.

[0019] Explanation of key component symbols: 200 - Flatbed material installation and detection system; 210 - Target recognition module; 220 - Image correction module; 230 - Image segmentation module; 240 - Material detection module. Detailed Implementation

[0020] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0021] The components of the embodiments of this application described and illustrated in the accompanying drawings can be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of this application provided in the drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] In the following text, the terms "comprising," "having," and their cognates, which may be used in various embodiments of this application, are intended only to indicate a particular feature, number, step, operation, element, component, or combination thereof, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations thereof, or adding the possibility of one or more combinations thereof. Furthermore, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0023] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of this application pertain. Terms (such as those defined in commonly used dictionaries) shall be interpreted as having the same meaning as in their contextual meaning in the relevant technical field and shall not be construed as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of this application.

[0024] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] The following describes the method for testing the installation of flat panel materials using specific examples.

[0026] Figure 1 A flowchart of a flat panel material installation and inspection method according to an embodiment of this application is shown. Exemplarily, the flat panel material installation and inspection method includes the following steps: S110: Obtain an image of the assembled flat panel, input the image of the flat panel into the target recognition model, and obtain the recognition result output by the target recognition model. The recognition result includes the outline of the flat panel in the image of the flat panel.

[0027] In this embodiment, the tablet assembly refers to the hardware assembly of a tablet computer, including a motherboard, screws, heat sink, ribbon cable, speaker, etc., and the tablet image refers to the image of the tablet assembly to be tested. The tablet image can be continuously acquired using a common USB (Universal Serial Bus) camera. Before acquiring the assembled tablet image, a stability check is performed. Formal testing is only triggered when the tablet under test enters the shooting area and meets the stability conditions.

[0028] In one embodiment, multiple initial images of the entire tablet and image parameters of each initial image of the entire tablet are acquired; it is determined whether the image parameters meet the corresponding preset threshold; and under the condition that the image parameters meet the corresponding preset threshold multiple times consecutively, an image of the entire tablet after assembly is acquired.

[0029] Exemplary, the image parameters include one or more of the following: the offset of the center position of the tablet outline in the initial tablet image from the center position of the standard tablet outline; the outline rotation angle between the tablet outline in the initial tablet image and the standard tablet outline; the rate of change of the area of ​​the tablet outline in the current frame of the initial tablet image from the area of ​​the tablet outline in the previous frame; and the sharpness of the initial tablet image.

[0030] The standard flatbed machine outline refers to the outline of the flatbed machine in an image taken when the machine is placed horizontally on a flat testing platform, with its front facing upwards, its long side parallel to the reference coordinate axis, and without any tilt, rotation, or obstruction. The coordinate system of the standard flatbed machine image is the template coordinate system. The template coordinate system is a pre-established standard flatbed machine image coordinate system for a specific machine model and testing station, used to standardize the spatial position of each installation location. When the flatbed machine model, assembly surface, or material layout changes, only a new template coordinate system and a testing template for each installation location need to be established; the entire testing process does not need to be rewritten.

[0031] This offset can be obtained by calculating the difference in the X-axis coordinate or the Y-axis coordinate between the center position of the flat panel outline in the initial flat panel image and the center position of the standard flat panel outline. The outline rotation angle refers to the clockwise or counterclockwise rotation angle of the actual flat panel outline detected in the current frame relative to the standard flat panel outline around its center, used to characterize attitude deviation. The area change rate is the percentage of the difference between the outline area of ​​the current frame and the outline area of ​​the previous frame relative to the area of ​​the previous frame, reflecting the dynamic changes in the size of the flat panel on the imaging plane due to scaling, distance changes, or deformation.

[0032] In one embodiment, the preset thresholds include one or more combinations of preset offset thresholds, preset contour rotation angle thresholds, preset area change rate thresholds, and preset image clarity thresholds. Stability determination can be achieved using any of the following methods, or combinations thereof: the offset of the initial flat panel images for N consecutive frames is less than a preset offset threshold, i.e., the offset meets the set preset offset threshold; the contour rotation angle of the initial flat panel images for N consecutive frames is less than a preset contour rotation angle threshold, i.e., the contour rotation angle meets the set preset contour rotation angle threshold; the area change rate of the initial flat panel images for N consecutive frames is less than a preset area change rate threshold, i.e., the area change rate meets the set preset area change rate threshold; and the image clarity of the initial flat panel images for N consecutive frames is greater than a preset image clarity threshold, i.e., the image clarity meets the set preset image clarity threshold. Wherein, N can be set to 2 to 10 frames according to the production line cycle time.

[0033] In this embodiment, the stability determination method can avoid taking images of the entire tablet before the operator has stabilized the tablet, thereby reducing misjudgments caused by motion blur and coordinate drift.

[0034] As an example, for a tablet computer image captured under stable conditions, a target recognition model is used to extract the contour of the tablet computer from the image, obtaining the outer contour region of the tablet computer. This target recognition model is a lightweight segmentation model for visual recognition of tablet computer images.

[0035] In one embodiment, the training process of the target recognition model includes: acquiring multiple images of the first tablet body, labeling the target in each first tablet body image to obtain a first dataset, wherein the target includes the outline of the tablet body, and the target also includes one or more combinations of the tablet body's camera and battery; and training the model based on the first dataset to obtain the target recognition model.

[0036] In this embodiment, image or video data of the tablet motherboard in the actual production line can be collected by the factory. Frames are extracted from the collected video, or a first image of the entire tablet is obtained directly from captured static images, thus forming training samples for the target recognition model. The first tablet image is annotated using a third-party open-source annotation tool to obtain a first dataset. The annotation categories include at least larger detection targets such as the tablet outline, camera, and battery. That is, larger detection targets in the image are preliminarily detected, while smaller materials are detected separately through further image segmentation, thereby improving the detection accuracy of smaller materials and avoiding missed detections of smaller materials.

[0037] In one implementation, the first dataset is divided into a training set and a validation set. An open-source YOLOv8 segmentation model can be used for training. This part does not modify the network structure of the open-source YOLOv8 segmentation model; instead, it performs transfer training on the open-source YOLOv8 segmentation model using the first dataset to obtain the object recognition model. The trained object recognition model is then exported for subsequent inference deployment on edge devices.

[0038] In another embodiment, the trained target recognition model can be deployed to a terminal device. The target recognition model runs on the terminal device in a local inference mode to identify the overall outline of the tablet computer under test, the camera, the battery, and other detection targets in real time.

[0039] In this embodiment, the open-source YOLOv8 segmentation model itself is not modified. Instead, the open-source YOLOv8 segmentation model trained by transfer learning is used to stably obtain the overall outline or key region information of the tablet, thereby providing a pre-input for subsequent image calibration and image segmentation.

[0040] S120: Correct the flat panel image based on the coordinate transformation matrix between the flat panel outline and the standard flat panel outline to obtain the corrected flat panel image.

[0041] As an example, in an actual production line, the position, angle, and tilt of the flat panel to be tested after being placed in the imaging area may not be completely consistent with the placement position of the standard flat panel. If calibration is not performed first, material offset can easily occur when directly cropping the target detection image based on the detection template, thus affecting the detection results.

[0042] In one embodiment, the coordinate transformation matrix is ​​a perspective transformation matrix that maps the pixel coordinates of the tablet outline in the tablet whole image to the coordinates in the standard tablet whole image coordinate system. Before correcting the tablet whole image according to the coordinate transformation matrix between the tablet whole image outline and the standard tablet whole image outline, the first coordinates of the target pixel points in the tablet whole image outline and the second coordinates of the target pixel points in the standard tablet whole image outline are obtained respectively. The coordinate transformation matrix for correcting the tablet whole image is calculated based on at least four sets of the first and second coordinates of the target pixel points.

[0043] Exemplary examples show that target pixels include one or more of the following: corner points, boundary points, and feature matching points of the overall tablet outline and the standard tablet outline. Corner points refer to the corner points of the circumscribed quadrilateral of the tablet extracted from the overall tablet outline. Boundary points refer to key boundary points extracted from the boundaries of the overall tablet outline, such as the corners of right-angle pads or the vertices of circular interface edges. Feature matching points refer to stable feature matching points between the standard tablet image and the overall tablet image, such as the corner of an L-shaped pad on the edge of a USB interface, the center of a symmetrical pin array on a chip package, or the center of a specific circular hole in a heat dissipation hole array.

[0044] Optionally, a lightweight segmentation model suitable for edge operation can be used to output the flat panel mask region, and then target pixels such as corner points, boundary points, and feature matching points of the flat panel's overall contour can be extracted from the mask region. In addition to using a lightweight segmentation model, edge extraction, contour fitting, corner detection, and minimum bounding rectangle methods can also be used to obtain target pixels.

[0045] In one embodiment, the coordinate transformation matrix can be obtained from the coordinates of at least four sets of target pixels. The coordinate transformation matrix can be expressed as: ; In the formula, Represents the coordinate transformation matrix. This represents the first element of the coordinate transformation matrix. This represents the second element of the coordinate transformation matrix. This represents the third element of the coordinate transformation matrix. This represents the fourth element of the coordinate transformation matrix. This represents the fifth element of the coordinate transformation matrix. This represents the sixth element of the coordinate transformation matrix. This represents the seventh element of the coordinate transformation matrix. This represents the eighth element of the coordinate transformation matrix. This represents the ninth element of the coordinate transformation matrix. (The engineering fixed constraints are also included.) .

[0046] As an example, let the coordinates of one set of target pixels in the overall outline of the tablet be (x, y), and their coordinates in the standard overall outline of the tablet be... The mapping relationship can then be established using a 3×3 coordinate transformation matrix H: ; in: ; ; ; In the formula, These are the homogeneous output coordinates obtained after perspective transformation. The horizontal homogeneous component corresponds to the standard flat panel whole-machine image coordinate system. Corresponding to the longitudinal homogeneous components, For scaled components.

[0047] Then we have: , ; Alternatively, for ease of solution, we can let h33 = 1. In this case, w = h31x + h32y + 1, leaving eight unknowns. , , , , , , , At least four sets of equations are needed to solve for the first and second coordinates of the target pixels. Two linear equations can be written for each set of the first and second coordinates of the target pixels. Four sets of points result in eight equations, which are just enough to solve for the eight unknown parameters.

[0048] In another embodiment, the coordinates of all pixels in the overall image of the tablet are transformed based on a coordinate transformation matrix to obtain a corrected image of the entire tablet. In this embodiment, each coordinate component of the homogeneous output coordinates is substituted into the equation. , In this process, the coordinates of each pixel in the flat panel image can be transformed to the coordinate system of the standard flat panel image to obtain the flat panel calibration image. This ensures that the flat panel calibration image and the standard flat panel image are under the same reference coordinates, and then the target detection image is extracted.

[0049] In one embodiment, the calibration image of the entire flat panel can be further normalized in brightness, color, or local contrast to reduce the impact of changes in ambient light. Through calibration and normalization processing, the originally randomly placed flat panels under test are uniformly mapped to the same template coordinate system, enabling stable material capture at each subsequent installation position, improving image quality, and enhancing the accuracy of material detection.

[0050] S130: Based on the detection template of each material in the flatbed machine, the calibration image of the flatbed machine is segmented to obtain multiple target detection images. Each target detection image is a region image of each material.

[0051] Exemplary, the inspection template is a pre-set template for each material at different mounting positions, tailored to different flat panel models, assembly surfaces, or workstations. It includes information such as the center coordinates of each material mounting position in the template coordinate system, the rectangular cutting dimensions, the material type, the corresponding inspection model number, and the adaptive confidence threshold. Its function is to guide precise region cropping and task distribution in the calibrated image, enabling refined and configurable quality inspection of materials at each mounting position. Material types for the flat panel assembly include, but are not limited to, screws, conductive cloth, acetate cloth, black Mylar, heat dissipation film, and antenna clips.

[0052] In one embodiment, the detection template includes the cutting dimensions and center coordinates of each material installation position in the standard flat panel whole machine image. The flat panel whole machine calibration image is segmented based on the cutting dimensions and center coordinates of each material installation position to obtain the target detection image.

[0053] In this embodiment, the center coordinates and cutting dimensions of the installation position of each material in the standard flat panel image are pre-set in the corresponding detection template. The cutting dimension is the width and height of a pre-defined rectangular cutting area for each material installation position, ensuring strict coverage of the material's complete shape. Different material installation positions can correspond to different cutting dimensions to accommodate materials of different shapes, such as screws, conductive cloth, camera modules, and speakers.

[0054] Since the coordinates of each pixel in the flat panel calibration image are transformed to the coordinate system of the standard flat panel image, the flat panel calibration image can be directly cut according to the center coordinates of the mounting position and the cutting size to obtain multiple target detection images containing a single material to be detected.

[0055] In one embodiment, each target detection image includes at least the material installation location number, material type, mid-coordinate and cutting size of the installation location where the material is located, material detection model number corresponding to the material type, and detection confidence threshold. For structurally complex areas, a layered approach of large cutting size plus sub-cutting size can be adopted. First, it is determined whether there are anomalies in the large cutting size area, and then the small cutting size is further subdivided for detection.

[0056] S140, input each target detection image into the corresponding material detection model according to the material type to obtain the detection result of the corresponding material in the plate.

[0057] In this embodiment, the problem of inconsistent coordinates caused by the random placement of the entire machine is first addressed, and then local identification is performed on the fixed material installation positions. This avoids false detections or missed detections caused by the offset of the plate position, changes in angle, or local misalignment when directly identifying on the entire image.

[0058] As an example, the material installation and inspection section adopts a local classification method. That is, instead of classifying the entire image uniformly, it determines whether the target material exists in the target inspection image corresponding to each preset installation position and whether the material is installed correctly.

[0059] Existing visual inspection solutions are mostly used for robot loading, material sorting, or general target recognition. They primarily address the question of what materials are present in a scene and their locations, rather than whether a specific fixed mounting position on the product is missing or incorrectly installed after assembly. The material detection model in this application is used to detect the presence of target materials and whether they are installed correctly, thereby solving the problem of identifying missing or incorrectly installed materials at fixed mounting positions after assembly.

[0060] In one embodiment, the training process of the material detection model includes: acquiring second images of multiple flatbed machines with correctly installed materials and third images of flatbed machines with abnormally installed materials, wherein the abnormal installation includes missing and / or misinstalled target materials; annotating and cropping the installation positions of the same material type in the second and third flatbed machine images to obtain a second dataset, wherein the second dataset includes positive and negative samples for each material type, wherein the positive samples are annotated image samples when the target material is correctly installed, and the negative samples are annotated image samples when the target material is missing and / or misinstalled; and training a binary classification model based on the positive and negative samples to obtain a material detection model for the corresponding material type.

[0061] In this embodiment, the factory collects a video of the actual assembly process, or a set of images showing materials installed correctly, materials not installed properly, missing materials, or abnormal conditions. Based on the type of material the factory is interested in, a third-party annotation tool is used to annotate the installation area of ​​each material, such as screws, acetate cloth, conductive cloth, etc. A script program then crops the installation area corresponding to each target material from the original image according to the annotation box position to form an annotated image sample.

[0062] In one implementation, a dataset can be created for labeled image samples of a certain type of material. Labeled image samples of correctly installed materials are placed in the positive sample set, while labeled image samples of materials in missing or incorrectly installed states are placed in the negative sample set. After dividing the positive and negative samples into training and validation sets, a pre-trained convolutional neural network is used for transfer learning training to output a material detection model corresponding to each material type. For newly added or changed materials, a small number of positive and negative samples can be collected, incremental training can be completed on a regular computer, and then the updated model can be distributed to the terminal device.

[0063] In this embodiment, image enhancement methods can be used to improve robustness during the classification model training phase. For example, scaling, translation, rotation, and flipping operations can be performed on the training samples to adapt to the effects of changes in lighting, angle, and local position on the production line. For each type of material to be detected, labeled samples are pre-constructed, including positive and negative samples. Based on this, a corresponding local classification and recognition material detection model is trained, which is used to subsequently determine whether materials at each installation position are missing or incorrectly installed.

[0064] In one embodiment, material detection models corresponding to each material type are deployed to the terminal device for performing local inference on the extracted target detection images. For multiple target detection image detection tasks, multi-threading or thread pooling can be used for parallel scheduling to shorten the detection time. Optionally, concurrent scheduling of multiple target detection images can also be achieved using asynchronous task queues, multi-processing, or lightweight inference engines.

[0065] For example, the detection template includes a detection confidence threshold for the material at each mounting location. The detection confidence threshold differs for the same type of material at different mounting locations, while the material detection model is the same for the same type of material at different mounting locations. For instance, if a camera is positioned directly above the center of the tablet, images taken from the edge areas of the tablet will be distorted, while images taken from the center area will have less distortion. Therefore, for the same type of material—such as screws installed at the edges of the tablet while others are installed in the center—the detection confidence threshold for the same type of material mounted in the center area will be higher, while the detection confidence threshold for the same type of material mounted at the edges will be lower.

[0066] In one embodiment, the detection result includes one of the following detection types: normal material installation and abnormal material installation. The detection result also includes the model detection confidence level corresponding to each installation position. For example... Figure 2 As shown, the method for testing the installation of flat-plate materials also includes the following steps: S150, compare the model detection confidence of each mounting position with the corresponding detection confidence threshold.

[0067] S160, under the condition that the model detection confidence level is greater than or equal to the detection confidence level threshold, the detection type output by the material detection model is taken as the material detection result of the installation position.

[0068] S170, if the model detection confidence level is less than the detection confidence level threshold, the material detection result at the installation position is determined to be pending re-inspection or installation abnormality.

[0069] In this embodiment, the detection confidence threshold is a preset template that reflects the prior weight of the imaging quality of the mounting position on the reliability of recognition, and the model detection confidence is the probability score of the specific detection type output by the material detection model for the current target detection image.

[0070] For each installation location, the material inspection model outputs a detection type and its corresponding detection confidence score. This score is compared to a preset detection confidence threshold for that location. Only when the detection confidence score is greater than or equal to the threshold is the result adopted as the final judgment; otherwise, it is considered unreliable, and a definitive conclusion is rejected, potentially triggering a re-inspection or manual intervention. This mechanism effectively suppresses misjudgments caused by local blurring, reflection, or obstruction, improving the accuracy of production line quality inspection.

[0071] In one embodiment, after the inspection is completed, the terminal device can directly output the target inspection result and simultaneously save the target inspection image, time information, product number or serial number information for subsequent traceability. This solves the problem that traditional manual inspection or simple visual judgment often does not automatically save inspection images and result records, resulting in a lack of effective traceability evidence when subsequent quality disputes occur.

[0072] In one implementation, such as Figure 3 and Figure 4 The image shown is a schematic diagram illustrating the installation and testing results of a certain type of flat panel material. Figure 3 and Figure 4 The green and red boxes in the diagram represent the target materials to be tested. Green indicates that the test has passed, meaning the materials are installed correctly, while red indicates that the test has failed, meaning there are missing or incorrect installations at the corresponding installation positions.

[0073] In one embodiment, for a small number of new materials, multiple standard detection images of the materials can be extracted from the standard flat panel whole machine image. The feature vector of each standard detection image and the feature vector of the target detection image are calculated. The feature vectors of the two are compared. If the feature vectors are inconsistent, the installation is judged to be incorrect. If they are consistent, the installation is judged to be correct, and there is no need to immediately retrain the model.

[0074] Figure 5 A schematic diagram of a flatbed material loading and inspection system 200 according to an embodiment of this application is shown. Exemplarily, the flatbed material loading and inspection system 200 includes: The target recognition module 210 is used to acquire an image of the assembled flat panel, input the image of the flat panel into the target recognition model, and obtain the recognition result output by the target recognition model. The recognition result includes the outline of the flat panel in the image of the flat panel.

[0075] In one embodiment, the target recognition module 210 is further configured to acquire multiple frames of initial flat panel images and image parameters of each frame of initial flat panel images; determine whether the image parameters meet the corresponding preset thresholds; and acquire the assembled flat panel image if the image parameters meet the corresponding preset thresholds multiple times consecutively; wherein the image parameters include one or more of the following: the offset between the center position of the flat panel outline in the initial flat panel image and the center position of the standard flat panel outline; the outline rotation angle between the flat panel outline in the initial flat panel image and the standard flat panel outline; the area change rate between the area of ​​the flat panel outline in the current frame of the initial flat panel image and the area of ​​the flat panel outline in the previous frame of the initial flat panel image; and the clarity of the initial flat panel image.

[0076] In one embodiment, the training process of the target recognition model includes: acquiring multiple images of the first tablet body, labeling the target in each first tablet body image to obtain a first dataset, wherein the target includes the outline of the tablet body, and the target also includes one or more combinations of the tablet body's camera and battery; and training the model based on the first dataset to obtain the target recognition model.

[0077] The image correction module 220 is used to correct the image of the flat panel according to the coordinate transformation matrix between the overall outline of the flat panel and the standard overall outline of the flat panel, so as to obtain the corrected image of the flat panel.

[0078] In one embodiment, the image correction module 220 is further configured to obtain the first coordinates of the target pixel points in the overall outline of the tablet and the second coordinates of the target pixel points in the standard overall outline of the tablet. The target pixel points include one or more of the corner points, boundary points, and feature matching points of the overall outline of the tablet and the standard overall outline. The module calculates a coordinate transformation matrix for correcting the overall image of the tablet based on the first and second coordinates of at least four sets of target pixel points.

[0079] In one embodiment, the image correction module 220 is further configured to perform coordinate transformation on the coordinates of all pixels in the whole image of the tablet based on the coordinate transformation matrix to obtain a corrected image of the whole tablet.

[0080] The image segmentation module 230 is used to segment the calibration image of the flat panel machine according to the detection template of each material of the flat panel machine to obtain multiple target detection images, and the target detection images are the region images of each material.

[0081] In one embodiment, the detection template includes the cutting dimensions and center coordinates of each material installation position in the standard flat panel whole machine image. The image segmentation module 230 is also used to perform image segmentation on the flat panel whole machine calibration image based on the cutting dimensions and center coordinates of each material installation position to obtain the target detection image.

[0082] The material detection module 240 is used to input the detection images of each target into the corresponding material detection model according to the material type, so as to obtain the detection results of the corresponding material in the plate.

[0083] In one embodiment, the training process of the material detection model includes: acquiring second images of multiple flatbed machines with correctly installed materials and third images of flatbed machines with abnormally installed materials, wherein the abnormal installation includes missing and / or misinstalled target materials; annotating and cropping the installation positions of the same material type in the second and third flatbed machine images to obtain a second dataset, wherein the second dataset includes positive and negative samples for each material type, wherein the positive samples are annotated image samples when the target material is correctly installed, and the negative samples are annotated image samples when the target material is missing and / or misinstalled; and training a binary classification model based on the positive and negative samples to obtain a material detection model for the corresponding material type.

[0084] In one embodiment, the detection template includes a detection confidence threshold for the material at each installation position, and the detection result includes one of the detection types: normal material installation and abnormal material installation. The detection result also includes the model detection confidence level corresponding to each installation position. The material detection module 240 is further configured to compare the model detection confidence level of each installation position with the corresponding detection confidence threshold; if the model detection confidence level is greater than or equal to the detection confidence threshold, the detection type output by the material detection model is used as the material detection result of the installation position; if the model detection confidence level is less than the detection confidence threshold, the material detection result of the installation position is determined as requiring re-inspection or installation abnormality; wherein, the detection confidence thresholds corresponding to the same type of material at different installation positions are different.

[0085] It is understood that the system in this embodiment corresponds to the flat material installation and detection method in the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0086] This application also provides a terminal device, exemplary of which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to enable the terminal device to perform the functions of the various modules in the above-described flat material installation and detection method or the above-described flat material installation and detection system.

[0087] The terminal device includes, but is not limited to, mobile phones, tablets, computers, and other devices. Based on the aforementioned method for detecting the installation of tablet materials, the terminal device can use a low-resolution camera to capture images of the entire tablet and detect the installation of tablet materials. It does not require industrial cameras, industrial control computers, special light sources, or complex mechanical positioning mechanisms. This solves the problems of high deployment costs, difficulty in scaling up and replicating traditional detection methods, and difficulty in rapid promotion across multiple production lines, and reduces the performance requirements of the terminal device.

[0088] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including at least one of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Network Processor (NP), Digital Signal Processor (DSP), Application-Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0089] Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), and Electrically Erasable Programmable Read-Only Memory (EEPROM). Memory is used to store computer programs, and the processor can execute these programs upon receiving execution instructions.

[0090] This application also provides a computer-readable storage medium for storing computer programs used in the aforementioned terminal devices. For example, the computer-readable storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0091] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that, as an alternative implementation, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] In addition, the functional modules or units in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0093] If a function is implemented as a software module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a smartphone, personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0094] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for detecting the installation of flat materials, characterized in that, include: Acquire an image of the assembled tablet, input the image into a target recognition model, and obtain the recognition result output by the target recognition model. The recognition result includes the outline of the tablet in the image. The image of the flat panel is corrected based on the coordinate transformation matrix between the outline of the flat panel and the outline of the standard flat panel to obtain a corrected image of the flat panel. The flatbed machine calibration image is segmented according to the detection template of each material in the flatbed machine to obtain multiple target detection images, wherein each target detection image is a region image of each material. Each of the target detection images is input into the corresponding material detection model according to the material type to obtain the detection result of the corresponding material in the plate; The detection template includes a detection confidence threshold for the material at each installation position; the detection result includes one of the following detection types: normal material installation and abnormal material installation; the detection result also includes the model detection confidence for each installation position; and the method further includes: The model detection confidence score for each mounting position is compared with the corresponding detection confidence threshold. If the detection confidence level of the model is greater than or equal to the detection confidence level threshold, the detection type output by the material detection model shall be used as the material detection result of the installation position. If the model detection confidence level is less than the detection confidence threshold, the material detection result at the installation location will be determined as needing re-inspection or installation abnormality. Among them, the detection confidence thresholds for the same type of material at different installation positions are different.

2. The method for detecting the installation of flat materials according to claim 1, characterized in that, The process of obtaining an image of the assembled tablet unit includes, prior to: Acquire multiple frames of initial flat panel images and image parameters for each frame of the initial flat panel image; Determine whether the image parameters meet the corresponding preset threshold, and if the image parameters meet the corresponding preset threshold multiple times consecutively, obtain the whole image of the tablet. The image parameters include one or more of the following: the offset between the center position of the tablet outline in the initial tablet image and the center position of the standard tablet outline; the outline rotation angle between the tablet outline in the initial tablet image and the standard tablet outline; the area change rate between the area of ​​the tablet outline in the current frame and the area of ​​the tablet outline in the previous frame; and the sharpness of the initial tablet image.

3. The method for detecting the installation of flat materials according to claim 1, characterized in that, Before correcting the tablet image based on the coordinate transformation matrix between the tablet's overall outline and the standard tablet's overall outline, the method further includes: The first coordinates of the target pixel in the overall outline of the tablet are obtained, and the second coordinates of the target pixel in the standard overall outline of the tablet are obtained. The target pixel includes one or more of the corner points, boundary points and feature matching points of the overall outline of the tablet and the standard overall outline of the tablet. The coordinate transformation matrix for correcting the whole image of the tablet is calculated based on the first and second coordinates of at least four sets of target pixels.

4. The method for detecting the installation of flat materials according to claim 3, characterized in that, The step of correcting the tablet image based on the coordinate transformation matrix between the tablet's overall outline and the standard tablet's overall outline includes: Based on the coordinate transformation matrix, the coordinates of all pixels in the whole image of the tablet are transformed to obtain the corrected image of the whole tablet.

5. The method for detecting the installation of flat materials according to claim 1, characterized in that, The training process of the target recognition model includes: Multiple images of the first tablet are acquired, and a recognition target is labeled in each of the first tablet images to obtain a first dataset. The recognition target includes the outline of the tablet and also includes one or more combinations of the tablet's camera and battery. The target recognition model is obtained by training the model based on the first dataset.

6. The method for detecting the installation of flat materials according to claim 1, characterized in that, The detection template includes the cutting dimensions and center coordinates of each material installation position in the standard flatbed machine image. The step of image segmentation of the corrected image of the flatbed machine based on the detection template for each material includes: The target detection image is obtained by segmenting the whole plate machine calibration image based on the cutting size and center coordinates of each material installation position.

7. The method for detecting the installation of flat materials according to claim 1, characterized in that, The training process of the material detection model includes: Acquire a second image of a flatbed machine with normal material installation and a third image of a flatbed machine with abnormal material installation, wherein the abnormal material installation includes missing and / or incorrectly installed target materials. The installation locations of the same material type are marked and cropped in the second and third flat panel whole images to obtain a second dataset. The second dataset includes positive and negative samples for each material type. The positive samples are marked image samples when the target material is installed correctly, and the negative samples are marked image samples when the target material is missing and / or misinstalled. A binary classification model is trained based on the positive and negative samples to obtain a material detection model for the corresponding material type.

8. A flat material installation and detection system, characterized in that, include: The target recognition module is used to acquire an image of the assembled tablet, input the image of the assembled tablet into the target recognition model, and obtain the recognition result output by the target recognition model. The recognition result includes the outline of the tablet in the image of the assembled tablet. The image correction module is used to correct the image of the flat panel according to the coordinate transformation matrix between the outline of the flat panel and the outline of the standard flat panel, so as to obtain a corrected image of the flat panel. The image segmentation module is used to segment the corrected image of the flat panel machine according to the detection template of each material of the flat panel machine to obtain multiple target detection images, wherein the target detection image is a region image of each material. The material detection module is used to input each of the target detection images into the corresponding material detection model according to the material type, so as to obtain the detection result of the corresponding material in the plate. The detection template includes a detection confidence threshold for the material at each installation position, the detection result includes one of the detection types: normal material installation and abnormal material installation, and the detection result also includes the model detection confidence for each installation position. The material detection module is also used to compare the model detection confidence level of each installation position with the corresponding detection confidence level threshold; If the detection confidence level of the model is greater than or equal to the detection confidence level threshold, the detection type output by the material detection model shall be used as the material detection result of the installation position. If the model detection confidence level is less than the detection confidence level threshold, the material detection result at the installation location is determined to be subject to re-inspection or an installation anomaly; wherein, the detection confidence level thresholds for the same type of material at different installation locations are different.

9. The flat material installation and detection system according to claim 8, characterized in that, The target recognition module is also used to acquire multiple frames of initial tablet images and image parameters of each frame of the initial tablet image; determine whether the image parameters meet the corresponding preset threshold, and acquire the tablet image if the image parameters meet the corresponding preset threshold multiple times consecutively. The image parameters include one or more of the following: the offset between the center position of the tablet outline in the initial tablet image and the center position of the standard tablet outline; the outline rotation angle between the tablet outline in the initial tablet image and the standard tablet outline; the area change rate between the area of ​​the tablet outline in the current frame and the area of ​​the tablet outline in the previous frame; and the sharpness of the initial tablet image.

10. A terminal device, characterized in that, The terminal device includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the flat material installation and detection method according to any one of claims 1-7.

Images