A defect detection method, device, computer equipment and storage medium

Abstract

The application discloses a defect detection method and device for pressing behavior of a reflecting sheet in a television screen processing process, a computer device and a storage medium. An online detection system based on high-frame-rate video acquisition and AI visual analysis is constructed, a vertical displacement sequence is extracted based on a tool center point trajectory, local minimum value identification and residence time determination are combined, at the same time, image pixel coordinates are mapped to actual physical space coordinates through a coordinate conversion matrix, accurate correspondence from visual space to workpiece space is realized, space attribution analysis of a pressing landing point is realized in combination with a preset area, fine coverage judgment of a real landing point and the preset area is realized, quantitative evaluation of coverage area and coverage ratio is realized through pixel-level coverage area accumulation and repeated deduplication calculation, and a defect category is determined in combination with a threshold. The application can realize real-time online detection and evaluation of the pressing condition of the reflecting sheet in the television screen processing process.

Description

Technical Field

[0001] This application belongs to the field of optical inspection, specifically relating to a defect detection method, device, computer equipment, and storage medium for determining whether the pressing behavior of the reflective sheet is in place during the processing of a television screen. Background Technology

[0002] In television screen manufacturing, the bonding quality of the reflective sheet directly determines the screen's optical display effect, making it a crucial step in the entire processing. Traditional inspection methods often rely on manual visual inspection or simple end-point image comparison. The main drawback of this method is that it only focuses on the final appearance after pressing, ignoring the dynamic force changes during the pressing process. Furthermore, it is highly susceptible to interference from complex lighting conditions in the workshop or operator obstruction, leading to serious misjudgments of incomplete or false pressing.

[0003] The core technical challenge in this field lies in the fact that the pressing action itself is a continuous and minute spatial movement process. In actual operation, the operating tool must undergo a complete descent, pause, and recovery phase. If the vertical displacement changes and specific pause times in this continuous movement cannot be captured at a high frequency, it is impossible to accurately determine whether the press was truly effective. This lack of motion feature capture can trigger a serious chain reaction; once the lowest point of the effective press cannot be confirmed, the actual coordinates of the press landing point cannot be extracted.

[0004] For example, on an actual production line, a worker might only lightly brush the edge of the reflector when operating tools, failing to reach the specified pressing depth and with insufficient dwell time. If the detection system misinterprets this invalid surface contact as a valid landing point, it will calculate a false area coverage rate when comparing the coordinates with the central main pressing area or the four critical corner pressing areas of the reflector. Even though there are large, uncovered areas on the reflector, the system might incorrectly record the landing point as completely acceptable, ultimately causing defective products to flow into the next process.

[0005] Therefore, accurately extracting the timing characteristics of continuous pressing actions to determine the effective landing point, and precisely matching it with the preset complex process area to calculate the actual coverage rate, has become a key issue in solving the problem of incomplete pressing of reflective sheets and quality traceability. Summary of the Invention

[0006] The purpose of this application is to provide a defect detection method, device, computer equipment, and storage medium to solve the technical problem that the prior art relies solely on endpoint detection, which cannot capture the dynamic characteristics of the pressing action, makes it difficult to accurately determine the effectiveness of the pressing action, and easily leads to misjudgments of leakage or false pressure.

[0007] To address the aforementioned technical problems, this application provides a defect detection method, employing the following technical solution: Acquire a continuous temporal video stream, extract the tool bounding box from the temporal video stream, and extract the vertical displacement sequence based on the trajectory of the center point of the tool bounding box to obtain the first displacement feature; Extract the minimum point of the first displacement feature. If the displacement change rate between adjacent frames before and after the minimum point is lower than the preset change threshold, extract the corresponding number of continuous frames to determine the dwell time. If the dwell time is greater than the preset time threshold, the spatial location corresponding to the minimum point is extracted to obtain the target low point of pressure. Extract the pixel position of the target's low point of pressure in the video frame, and obtain the actual landing point coordinates by mapping the pixel position through a coordinate transformation matrix; Obtain a pre-established preset area and calculate the inclusion relationship between the actual landing point coordinates and the preset area; If the actual landing point coordinates are within the preset area, the corresponding unit pixel area is accumulated to obtain the coverage area, and the coverage ratio is calculated. If the coverage ratio is less than the preset ratio threshold, a leakage pressure defect is determined, and a first quality judgment result is generated. The first quality judgment result and the vertical displacement sequence are classified to determine the defect identification category.

[0008] To address the aforementioned technical problems, this application also provides a defect detection device, which employs the following technical solution: The video stream processing module is used to acquire continuous temporal video streams, extract tool bounding boxes from the temporal video streams, and extract vertical displacement sequences based on the trajectory of the center point of the tool bounding boxes to obtain the first displacement feature. The minimum point module is used to extract the minimum point of the first displacement feature. If the displacement change rate between adjacent frames before and after the minimum point is lower than the preset change threshold, the corresponding number of continuous frames is extracted to determine the dwell time. The low-point press module is used to extract the spatial location corresponding to the minimum point if the dwell time is greater than a preset time threshold, and obtain the target low-point press. The coordinate mapping module is used to extract the pixel position of the target's low point of pressure in the video frame, and obtain the real landing point coordinates by mapping the pixel position through the coordinate transformation matrix; The region identification module is used to obtain a pre-established preset region and calculate the inclusion relationship between the actual landing point coordinates and the preset region. The region coverage module is used to accumulate the corresponding unit pixel area to obtain the coverage area and calculate the coverage ratio if the actual landing point coordinates are located within the preset area. The leakage pressure defect module is used to determine the existence of leakage pressure defect if the coverage ratio is less than a preset ratio threshold, and generate a first quality judgment result. The defect identification module is used to classify the first quality judgment result and the vertical displacement sequence to determine the defect identification category.

[0009] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: A computer device includes a memory and a processor, the memory storing computer-readable instructions, the processor executing the computer-readable instructions to implement the steps of the defect detection method as described in any of the preceding claims.

[0010] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below: A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, implement the steps of the defect detection method as described in any one of the preceding descriptions.

[0011] Compared with the prior art, the embodiments of this application have the following main advantages: This application discloses a method, device, computer equipment, and storage medium for detecting defects in the pressing behavior of reflective sheets during television screen manufacturing. By constructing an online detection system based on high frame rate video acquisition and AI visual analysis, it achieves full-process dynamic monitoring and quantitative analysis of the pressing behavior of reflective sheets during manufacturing. Based on the tool center point trajectory, it extracts the vertical displacement sequence and combines local minimum value identification and dwell time determination to achieve accurate identification of effective pressing actions. At the same time, it maps the image pixel coordinates to the actual physical space coordinates through a coordinate transformation matrix, achieving a precise correspondence from visual space to workpiece space. Furthermore, it combines a preset area model to perform spatial attribution analysis of the pressing point, thereby achieving a refined coverage judgment of the actual pressing point and the preset area. Through pixel-level coverage area accumulation and duplicate deduplication calculation, it achieves quantitative evaluation of coverage area and coverage ratio, enabling the pressing integrity to be objectively expressed numerically. Combined with threshold judgment, it achieves automatic identification of defects such as missed pressing, false pressing, and incomplete pressing. Overall, this application enables real-time online detection, precise spatial positioning, and quantitative assessment of the pressing of reflective sheets during television screen manufacturing, significantly improving the automation level and consistency control of the detection process, and enhancing the traceability and quality stability of the production process. Attached Figure Description

[0012] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 An exemplary system architecture diagram is shown, in which this application can be applied; Figure 2 A flowchart of one embodiment of the defect detection method according to this application is shown; Figure 3 It shows Figure 2 A flowchart of an embodiment of step S206; Figure 4 A schematic diagram of one embodiment of the defect detection apparatus according to this application is shown; Figure 5 It shows Figure 4 A schematic diagram of a structure of an embodiment of the central area coverage module 406; Figure 6 A schematic diagram of the structure of one embodiment of a computer device according to this application is shown. Detailed Implementation

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0015] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0017] like Figure 1As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0018] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0019] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0020] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0021] It should be noted that the defect detection method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the defect detection device is generally set in the server / terminal device.

[0022] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative; the system can have any number of terminal devices, networks, and servers depending on implementation needs.

[0023] Continue to refer to Figure 2 A flowchart of an embodiment of the defect detection method according to this application is shown. The defect detection method includes the following steps: S201, acquire a continuous temporal video stream, extract the tool bounding box in the temporal video stream, and extract the vertical displacement sequence based on the trajectory of the center point of the tool bounding box to obtain the first displacement feature; Specifically, in this step, the system continuously captures frames of the work area using an industrial camera deployed above the workstation, forming a temporally ordered image sequence data stream. The video stream is then decoded and frame extracted to ensure the temporal continuity of the analysis. In each frame, a target detection model identifies the work tool or operator's hand, outputting the corresponding target's bounding box information, and calculating the center point coordinates based on the bounding box's geometric center. Subsequently, through target association and tracking algorithms across multiple frames (such as a Kalman filter-based multi-target tracking method), a temporal trajectory sequence of the center point is constructed. Based on this, the two-dimensional coordinate sequence is projected or decomposed according to the vertical direction in the image coordinate system, extracting the vertical coordinate changes to form a time-varying vertical displacement sequence. This sequence is further smoothed or noise suppressed to obtain a stable first displacement feature data structure.

[0024] S202, extract the minimum point of the first displacement feature. If the displacement change rate between adjacent frames before and after the minimum point is lower than the preset change threshold, extract the corresponding number of continuous frames to determine the dwell time. Specifically, the process begins by detecting local extrema in the first displacement feature sequence. Local minima are identified in the displacement curve using a sliding window or differential calculation method. These minima typically correspond to the lowest position during the tool's downward movement. Then, centered on this minima, the rate of change of displacement per unit time or between adjacent frames is calculated in both forward and backward time directions. For example, the rate of change is obtained using first-order differential or slope calculation methods and compared with a preset threshold. When the rate of change in both directions before and after the minima is lower than the threshold, the position is considered to be in a relatively stable state. The number of consecutive frames in which this stable state persists is further counted, and time conversion is performed using the video capture frame rate parameter to obtain the corresponding dwell time parameter, which characterizes the tool's dwell behavior in the lowest point region.

[0025] S203, if the dwell time is greater than the preset time threshold, extract the spatial location corresponding to the minimum point to obtain the target low point of pressure; Specifically, based on the dwell time calculated in the preceding steps, a conditional judgment is made. When the dwell time exceeds a preset threshold, the system considers the action corresponding to the minimum point to have valid pressing characteristics. At this point, the spatial coordinate information, including the coordinates of the tool center point or the lowest point of the detection box, is extracted from the frame sequence corresponding to the minimum point, and its two-dimensional position in the image coordinate system is determined. Simultaneously, the specific frame number of the minimum point is located by combining the temporal dimension, thereby achieving spatiotemporal joint localization. Subsequently, filtering or outlier removal processing can be performed on this spatial position to avoid offset errors caused by detection jitter or occlusion. Finally, the stable corresponding spatial coordinates are defined as the target pressing low point, which serves as the basic input data for calculating the true landing point and region matching analysis.

[0026] If the dwell time does not exceed the preset threshold, the pressing action is considered not to have met the requirements for effective pressing. The current minimum point is directly filtered out and not entered into the subsequent landing point analysis process. Instead, the process continues to traverse and process the next candidate minimum point in the vertical displacement sequence.

[0027] S204: Extract the pixel position of the target's low point of pressure in the video frame, and obtain the real landing point coordinates by mapping the pixel position through a coordinate transformation matrix. Specifically, the pixel-level coordinates of the target's low-pressure point are first precisely extracted from the corresponding video frame. These coordinates are derived from the detection model output or trajectory interpolation results. Due to perspective distortion or installation angle deviations between the image coordinate system and the actual physical space, a coordinate transformation matrix is ​​needed for spatial mapping. This transformation matrix is ​​typically calculated based on camera calibration results or homography matrices and describes the geometric mapping relationship between the pixel plane and the actual workpiece plane. During processing, the pixel coordinates are input into the coordinate transformation model, and after matrix multiplication or perspective transformation, the corresponding actual physical coordinate values ​​are obtained. These true landing point coordinates are usually represented in the workpiece coordinate system or world coordinate system and used for subsequent region matching and spatial analysis, thereby achieving a unified transformation from image space to physical space.

[0028] S205, Obtain the pre-established preset area and calculate the inclusion relationship between the actual landing point coordinates and the preset area; Specifically, the system first loads a predefined standard operating area based on the product model or process configuration. This area is typically described as a polygon or rectangle and includes the boundary coordinates of multiple functional zones, such as the central area, edge areas, or key inspection areas. The system imports the actual landing point coordinates into this area model and calculates the inclusion relationship using geometric judgment algorithms, such as using the ray method or boundary equation method to determine whether the point is located inside the polygon. If the preset area has multiple sub-areas, each sub-area needs to be determined separately to identify the specific functional zone to which the landing point belongs. Simultaneously, by combining area priority rules, the system can handle overlapping areas, thereby outputting the spatial inclusion relationship result between the actual landing point and the preset area.

[0029] S206, If the actual landing point coordinates are within the preset area, the corresponding unit pixel area is accumulated to obtain the coverage area, and the coverage ratio is calculated. Specifically, after confirming that the actual landing point coordinates are within the preset area, the system generates a corresponding coverage pixel range centered on this landing point based on a preset spatial expansion model, such as a circular area based on a fixed radius or a pixel area based on morphological dilation. Then, each pixel within this area is statistically analyzed, and converted into a corresponding unit physical area value according to pixel calibration parameters. During multiple presses, the coverage areas of each press are accumulated, and duplicate coverage areas are removed to avoid double counting. Finally, the ratio of the accumulated coverage area to the total area of ​​the preset area is calculated to obtain a coverage ratio parameter, which characterizes the completeness of the target area covered by the pressing action.

[0030] If the actual landing point coordinates are outside the preset area, the pressing action is determined to be an erroneous operation outside the area. The coverage area of ​​the preset area is not accumulated, and the current accumulated coverage area result is directly retained.

[0031] S207, If the coverage ratio is less than the preset ratio threshold, it is determined that there is a leakage pressure defect and the first quality judgment result is generated. Specifically, the calculated coverage ratio is compared with a threshold. When the coverage ratio is lower than a preset standard threshold, the system determines that the current process has insufficient coverage. This judgment process usually combines the statistical results of multiple presses for comprehensive analysis to avoid the influence of single errors on the final judgment. After an anomaly is determined, the system generates a corresponding quality judgment result data structure, which includes information such as defect type identifier, occurrence time, corresponding workstation information, and relevant press sequence characteristic parameters. At the same time, this result can be used for subsequent alarm, recording, or traceability analysis modules, and serves as input data for classification processing to achieve preliminary identification and marking of leakage defects.

[0032] If the coverage ratio is greater than or equal to the preset ratio threshold, it is determined that the current preset area has completed the pressing operation that meets the process requirements and there is no leakage defect. The quality status of the current area is marked as qualified, and the first quality judgment result with the corresponding qualified label is generated, and the process enters the subsequent classification process.

[0033] S208, classify the first quality judgment result and the vertical displacement sequence to determine the defect identification category.

[0034] Specifically, the previously generated quality assessment results are fused and analyzed with the complete vertical displacement time series. Multidimensional data is comprehensively processed through feature extraction and classification models. For example, time series features such as peak and valley characteristics, number of dwell times, and displacement fluctuation amplitude can be extracted from the displacement sequence and jointly modeled with coverage ratio and regional distribution characteristics from the quality assessment results. Subsequently, different defect patterns are distinguished using rule-based classification methods or machine learning classification models, such as leakage pressure, false pressure, incomplete positioning, or path deviation, thereby outputting the final defect identification category result, achieving detailed identification and classification of different abnormal behaviors.

[0035] The specific classification process is as follows: The first quality judgment result is converted into structured category features. If the first quality judgment result is qualified, the category feature is marked as 0; if the first quality judgment result is that there is a leakage pressure defect, the category feature is marked as 1. Then, multi-dimensional statistical features are extracted from the vertical displacement sequence, including features of multiple dimensions such as the extreme value difference, average rate of change, total residence time, and number of extreme points of the displacement sequence. The above statistical features are concatenated with the structured category features to form a fused feature vector. The fused feature vector is input into the pre-trained classification model, which outputs the probability values ​​of different defect categories. The category corresponding to the maximum probability is selected as the final defect identification category, and the defect classification output is completed.

[0036] This embodiment provides a defect detection method based on AI vision. This method is applied to the online detection scenario of the reflective sheet pressing station. It is mainly used to monitor the pressing action of the operator on the reflective sheet in real time during the production process, and to automatically determine and classify defects such as incomplete pressing, inadequate pressing, or false pressing.

[0037] In actual deployment, a high-frame-rate industrial camera is first installed directly above the workstation. The camera is fixed at a preset height using a rigid bracket, ensuring its field of view completely covers the reflective sheet working area. After the system starts, the industrial camera continuously acquires video streams of the working area at a fixed frame rate, forming a continuous time-series image sequence. This video stream is then decoded and preprocessed in real time by an edge computing unit, including image denoising, distortion correction, and brightness equalization, thereby ensuring the stability and accuracy of target detection.

[0038] The system inputs a deep learning object detection model, such as a detection network based on YOLO or a similar structure, into each frame of the image to identify the pressing tool or the operating hand in the image and output the corresponding object detection box. The detection box is usually represented in the form of a rectangle and includes the center point coordinates (x, y) of the object. The system associates the detection results in consecutive frames and continuously tracks the object trajectory using a Kalman filter algorithm, thus forming a temporal trajectory sequence of the tool's center point. Based on this, the trajectory points are arranged in chronological order in the image coordinate system, and the displacement change in the vertical direction (y-axis direction) is extracted to obtain a vertical displacement sequence. This sequence is used to characterize the up-and-down movement trend of the tool during the pressing process and serves as the basic feature for subsequent action recognition.

[0039] The system performs local extremum analysis on the vertical displacement sequence, identifying potential minima using a sliding window approach. These minima typically correspond to the lowest position in a pressing action. Then, using this minima as the center, the system calculates the rate of change of displacement between adjacent frames, for example, by obtaining the trend through first-order difference or slope methods. When the rate of change of displacement between adjacent frames before and after the minima is lower than a preset threshold, the position is considered to be in a stable dwell state. The system counts the number of consecutive frames corresponding to this stable state and calculates the dwell time Δt based on the frame rate parameter, thereby determining whether the pressing action possesses effective dwell characteristics.

[0040] When the dwell time Δt exceeds a preset time threshold (e.g., 150ms~300ms), the system considers this minimum point to correspond to a valid pressing action. At this point, the spatial position of the tool center point is extracted from the corresponding frame, and its specific position in the workpiece coordinate system is determined by combining it with trajectory information, thus obtaining the target pressing low point. This low point reflects the actual contact point of the pressing action and is a key input for subsequent impact point analysis and region matching.

[0041] The system extracts the pixel coordinates of the target's lowest point of pressure in the current image frame and performs spatial mapping using a pre-calibrated coordinate transformation matrix. This matrix, constructed based on camera intrinsic and extrinsic parameter calibration results, is used to convert image pixel coordinates to actual physical plane coordinates. After the matrix transformation, the true landing point coordinates are obtained, which correspond to the actual spatial position of the workpiece.

[0042] The system loads a pre-established product process standard area model. This model is configured according to different product models and typically includes a central main pressure area, edge pressure areas, and four key corner pressure areas, storing their boundary coordinates in the form of polygonal regions. The system inputs the actual landing point coordinates into this area model and uses a point-within-polygon determination algorithm (such as ray casting or vector casting) to determine whether the point falls within the preset area, and outputs the region containment relationship result.

[0043] When the landing point is determined to be within the preset area, the system generates a local coverage area centered on that landing point. This area can be constructed based on a fixed-radius circular model or a pixel expansion model. The system iterates through and counts the pixels within the coverage area, and calculates the actual physical area corresponding to each pixel based on pixel calibration parameters. For multiple presses, the system overlays and counts all coverage areas, and removes duplicate coverage areas to obtain the cumulative coverage area. Simultaneously, the cumulative coverage area is compared to the total area of ​​the preset area to obtain the coverage ratio, which is used to measure the completeness of the coverage of the pressed area.

[0044] The system compares and analyzes the coverage ratio against a preset threshold. When the coverage ratio is lower than the set standard, such as failing to meet the 95% or 100% requirement, the system determines that there is a leakage pressure or insufficient coverage defect and generates a first quality judgment result. This result includes the defect type, occurrence time, workstation number, and corresponding pressing trajectory characteristic information, and is simultaneously sent to the edge control unit and the host computer system for alarm prompts and quality traceability.

[0045] The system further integrates and analyzes the initial quality assessment results with the vertical displacement sequence, and classifies defects by constructing a multi-dimensional feature vector. Specifically, this includes extracting features such as the number of peaks, dwell times, descent amplitude, and trajectory stability from the displacement sequence, while also combining coverage features for comprehensive judgment. The system can use rule-based classification methods or lightweight machine learning models to identify defect types, classifying them into categories such as leakage pressure, false pressure, incomplete positioning, or path deviation, and outputting the final defect identification results.

[0046] Throughout the implementation process, the system can also display the operation trajectory in real time through a visualization module. Specifically, the tool movement trajectory is overlaid on the video screen in a time series format, with the vertical trajectory represented in blue, the horizontal trajectory in green, and the effective pressing points marked with red dots. Simultaneously, the covered area is compared with the standard area, and the coverage ratio and detection status information are updated in real time, thereby achieving full-process visual monitoring and traceability recording of the operation.

[0047] Through the above implementation methods, this method can realize dynamic detection of the entire process of pressing the reflective sheet, from action recognition, dwell analysis, landing point calculation to coverage assessment and defect classification, forming a complete closed-loop detection system, which is suitable for high-precision industrial online detection scenarios.

[0048] Further, the steps of acquiring a continuous temporal video stream, extracting tool bounding boxes from the temporal video stream, and extracting a vertical displacement sequence based on the trajectory of the center point of the tool bounding box to obtain the first displacement feature specifically include: The pressing operation process is continuously captured by an industrial camera installed above the workstation, resulting in a time-series video stream containing the movement of the pressing tool. Perform target detection on each frame of the time-series video stream, identify the bounding box of the tool or the hand that is pressing, and extract the coordinates of the center point of the bounding box. The coordinates of the center point in each frame are correlated and tracked in chronological order to form a sequence of tool center point trajectories, and the vertical coordinate changes in the center point trajectory sequence are extracted. A vertical displacement sequence is constructed based on the vertical coordinate change, and a time series analysis is performed on the vertical displacement sequence to extract the compression depth, displacement change trend curve and dwell characteristics, thereby obtaining the first displacement feature.

[0049] In this embodiment, the industrial camera is calibrated and distortion corrected to improve imaging geometric accuracy; high frame rate acquisition ensures fine-grained expression of motion, and target detection models such as YOLO are used to achieve stable positioning of tools / hands; a target association tracking algorithm is introduced between multiple frames to continuously correct the center point trajectory, and Kalman filtering is used to smooth trajectory noise; simultaneously, pixel coordinates are mapped to a unified reference coordinate system based on camera intrinsic and extrinsic parameters, thereby converting vertical pixel displacement into actual physical displacement; further, the displacement sequence is segmented to identify the pressing phase, holding phase, and rebound phase, and multi-dimensional features such as pressing depth, instantaneous velocity, acceleration, and dwell time are calculated to form a structured first displacement feature vector for behavior determination and quality analysis.

[0050] In one specific embodiment of this application, in the scenario of detecting the pressing behavior of reflective sheet during the processing of a television screen, the time-series images captured by the industrial camera are processed accordingly, such as target detection, trajectory tracking, coordinate transformation, and landing point analysis, to automatically identify the position and dwell state of each pressing action. Then, through coverage calculation and multi-feature fusion classification, the defect judgment result is automatically output without the need for manual intervention for re-inspection. This effectively solves the problems of high missed and false judgment rates and lack of traceability in traditional manual inspection. At the same time, it can adapt to the process configuration requirements of different product models and can be flexibly integrated into existing automated production lines to improve inspection efficiency and quality control level.

[0051] Through the above steps, high-precision quantification and stable recognition of the pressing process can be achieved, improving the ability to resist obstruction and adapt to complex environments, and enhancing the reliability and consistency of detection.

[0052] Further, the step of extracting the minimum point of the first displacement feature, and if the displacement change rate between adjacent frames before and after the minimum point is lower than a preset change threshold, then extracting the corresponding number of continuous frames to determine the dwell time, specifically includes: The vertical displacement sequence corresponding to the first displacement feature is traversed in time, and local minimum points with displacement values ​​lower than adjacent time points are extracted as candidate low pressing points. Calculate the displacement change rate between adjacent frames before and after the candidate press low point, and compare the displacement change rate with a preset change threshold. If the displacement change rate before and after the candidate low point is lower than the preset change threshold, it is determined that the pressing action corresponding to the candidate low point is in a stable state. The number of consecutive video frames corresponding to a stable dwell state is counted, and the dwell time corresponding to the candidate low press point is obtained by combining the video frame rate.

[0053] In this embodiment, the system first performs frame-by-frame temporal scanning of the vertical displacement sequence based on the first displacement feature extracted from the continuous video stream, and analyzes the local change trends in the sequence through a sliding window mechanism. During this process, the system uses the current frame as a baseline and compares it with the displacement values ​​of adjacent frames before and after it, identifying data points that satisfy the condition "the current displacement value is less than the displacement value of the adjacent time point" as candidate local minima, used to characterize the lowest possible position state of the tool during pressing. For each candidate minima, the system further quantifies the displacement change within a certain time window before and after it, calculating the difference or rate of change between adjacent frames to obtain the corresponding displacement change rate curve, and comparing this rate of change with a preset stability threshold. When the rate of change of a candidate minima in both its forward and backward time intervals is lower than the set threshold, it indicates that the position is in a relatively stable low-position dwell state. At this time, the system marks the candidate point as a valid low-point candidate and enters the stable state determination process. Based on this, the system performs continuity detection on the frame sequence that maintains the stable state. By counting the number of frames that continuously meet the conditions, the system obtains the duration of the stable state's continuous frames. This duration is then combined with the video capture frame rate parameter for time conversion to calculate the corresponding dwell time parameter. This dwell time characterizes the degree to which the pressing action lingers in the lowest point region. It can also distinguish between momentary touches and valid pressing actions, avoiding misjudgments caused by brief jitters or rapid passage, thereby improving the stability and reliability of action recognition.

[0054] In the manufacturing process of television screens, reflective sheets typically require manual or semi-automatic multi-point pressing at bonding or pressing stations to ensure complete adhesion to the screen structure without air bubbles or gaps. During this process, an industrial camera continuously captures video streams of the operator's handheld pressing tool. The system extracts the center point trajectory of the pressing tool through target detection, forming a vertical displacement sequence. For example, when an operator presses the center area of ​​a reflective sheet on a television screen, the tool rapidly descends from above to the screen surface, enters a brief stable holding phase after contacting the reflective sheet, and then slowly rebounds. During this process, the displacement sequence shows a significant drop at the moment of contact, forming a relatively stable "plateau area" near the lowest point. Within this plateau area, the system identifies multiple consecutive frames where the rate of displacement change is significantly lower than a preset threshold, thus classifying this area as a stable dwell state and extracting the corresponding number of consecutive frames, such as a stable holding process lasting approximately 180ms. This duration corresponds to the operator's action characteristic of completing an effective press at that point. If, during the processing of another television screen, the operator only lightly touches the surface of the reflective sheet and then quickly lifts it, although a brief minimum value may occur, the system will not determine it as a valid pause due to insufficient frame count and a high rate of change. This avoids misidentifying it as a completed pressing action and achieves refined recognition and control of the pressing behavior of the reflective sheet.

[0055] Through the above steps, the stability of the lowest point of the pressing action and the dwell time can be accurately identified, effectively improving the accuracy and anti-interference ability of effective pressing judgment.

[0056] Furthermore, the steps of obtaining a pre-established preset region and calculating the containment relationship between the actual landing point coordinates and the preset region specifically include: Based on the current product model, a corresponding standard pressing area is pre-established, and the standard pressing area is divided into a central main pressing area, an edge pressing area, and four corner key pressing areas, and the boundary coordinate information corresponding to each pre-established area is obtained. Read the actual landing point coordinates and map them to the corresponding region coordinate system; Based on the spatial relationship between points and regions, the coordinates of the actual landing point are calculated to determine whether they are within the boundary range of the corresponding preset region, thereby obtaining the region inclusion relationship result. The region boundary range is determined based on the boundary coordinate information corresponding to each preset region. If the actual landing point coordinates are within the preset area boundary, the corresponding area is marked as a valid coverage area, and the corresponding area type and coverage count are recorded.

[0057] In this embodiment, the system first loads the corresponding standard pressing area model of the reflective sheet into a pre-configured database according to the process requirements of different TV screen models. This model is generated based on product design drawings or calibration data and is uniformly expressed based on the workpiece coordinate system. The standard area is further divided into multiple functional sub-areas, including a central main pressing area to ensure the core bonding quality, an edge pressing area to enhance the edge bonding reliability, and four corner key pressing areas for structural stability control. Each area is defined by polygonal or regular geometric boundaries and stores the corresponding set of boundary coordinates. During the actual detection process, the system obtains the actual landing point coordinates after mapping through a coordinate transformation matrix. These coordinates are derived from the detection results of the pressing low point in the image space and have been converted from pixel coordinates to physical space coordinates. Subsequently, the system uses a point-polygon spatial relationship judgment algorithm to perform regional classification analysis on the actual landing point coordinates. For example, it uses the ray method to determine whether it is located inside a certain closed area, or calculates its relative positional relationship through boundary equations, thereby determining whether the point falls within the preset area. When there are multiple sub-areas, the system will also perform matching judgment according to regional priority or process rules to avoid ambiguous identification caused by overlapping areas. After completing the spatial inclusion relationship calculation, the system marks the specific area type where the landing point is located and simultaneously records the number of times the area is covered for that press.

[0058] The central main pressing area, edge pressing area, and four key corner pressing areas refer to different functional pressing areas defined according to product process requirements, corresponding to the bonding quality requirements of different positions on the reflective sheet. The central main pressing area is the core pressing range in the center of the reflective sheet; the bonding quality in this area directly affects the overall flatness of the reflective sheet and the control of central air bubbles, making it a core area for quality control. The edge pressing areas are distributed around the edges of the reflective sheet, mainly used to eliminate gaps between the reflective sheet edges and the screen substrate, avoiding problems such as edge warping and delamination. The four key corner pressing areas are generally located at the four corners of the reflective sheet. The corners are where the reflective sheet is most prone to edge warping and air bubble accumulation, therefore requiring separate key control areas to ensure proper pressing coverage. These three types of areas together constitute a complete standard area system for the reflective sheet pressing process, adapting to the bonding quality control needs of different positions.

[0059] In a specific embodiment of this application, after obtaining the boundary coordinate information corresponding to each preset region, the boundary range of each region is generated using the boundary coordinate information corresponding to each preset region. Specifically, the boundary can be appropriately expanded outward by pixel dilation or fixed radius expansion to adapt to the coverage characteristics of the actual operation of the pressing tool, avoiding the misjudgment of valid landing points as outside the region due to small errors in coordinate transformation. After the boundary range is generated, different judgment priorities can be set for different regions. For example, when the landing point falls into the overlapping range of the four corner key pressing areas and the edge pressing areas at the same time, the landing point is preferentially assigned to the four corner key pressing areas for recording, ensuring the accuracy of coverage statistics for key control areas. The generated boundary range will be directly used for landing point inclusion relationship determination, providing an accurate regional basis for coverage calculation and defect identification.

[0060] Through the above steps, precise area matching and coverage statistics of the pressing point can be achieved, improving the accuracy of spatial positioning and process consistency judgment.

[0061] Further, please refer to Figure 3 If the actual landing point coordinates are within the preset area, the corresponding unit pixel area is accumulated to obtain the coverage area, and the coverage ratio is calculated. This process specifically includes: S301, when the actual landing point coordinates are within the preset area, establish the corresponding press coverage pixel area with the actual landing point coordinates as the center; S302, extract the unit pixels in the pressed and covered pixel area, and accumulate the pixel area corresponding to each unit pixel to obtain the coverage area corresponding to the current pressing action; S303, summarizes and counts the coverage area corresponding to continuous pressing actions, and removes duplicate pixel areas to obtain the cumulative coverage area of ​​the target area. S304, obtain the total area corresponding to the preset area, and calculate the coverage ratio based on the cumulative coverage area and the total area; S305 compares the coverage ratio with the preset coverage standard and outputs the corresponding area coverage result.

[0062] In this embodiment, when the system confirms that the actual landing point coordinates are within a preset area, it first constructs a local pressing coverage model with this landing point as the geometric center. This model typically uses a circular neighborhood with a fixed radius or a pixel expansion region generated based on morphological dilation operations to simulate the effective range of action generated by the contact between the tool and the reflective sheet during actual pressing. Within this coverage area, the system performs point-by-point traversal analysis on all pixels and maps each pixel to its corresponding physical size or equivalent area unit in conjunction with camera calibration parameters, thereby forming a set of unit pixel areas. During a single pressing operation, the system accumulates this set to obtain the coverage area corresponding to that pressing action. Simultaneously, during multiple consecutive pressing operations, the system performs temporal superposition processing on the coverage areas generated by each pressing and removes duplicate pixel regions that overlap spatially, for example, by using set operations or raster marking to avoid duplicate statistics, thereby obtaining the true cumulative coverage area of ​​the target area. Based on this, the system further obtains the standard total area of ​​the preset area, which is derived from the area definition parameters in the product process model, and calculates the ratio with the cumulative coverage area to obtain the coverage ratio. This coverage ratio reflects the overall coverage of the target area by the actual pressing action and can be further used to distinguish between complete and insufficient coverage. Simultaneously, the system compares the calculated coverage ratio with a preset process standard threshold and outputs the corresponding area coverage determination result.

[0063] The above steps enable accurate quantification of the coverage area of ​​the pressing zone and deduplication of duplicate coverage, thereby improving the reliability of coverage ratio assessment.

[0064] Further, the steps of extracting unit pixels within the pressed and covered pixel area, and accumulating the pixel areas corresponding to each unit pixel to obtain the coverage area corresponding to the current pressing action, specifically include: Generate the corresponding press coverage pixel area based on the actual landing point coordinates and the preset press radius, and determine the boundary range corresponding to the press coverage pixel area; Iterate through each pixel within the pressed and covered pixel area and extract the coordinate information of the unit pixel within the boundary range; The actual pixel area corresponding to each unit pixel is obtained based on the image calibration parameters, and the actual pixel areas are accumulated and calculated. Output the cumulative pixel area corresponding to the current press action as the coverage area corresponding to the current press action.

[0065] In this embodiment, after the system obtains the actual landing point coordinates and the preset pressing radius, it first constructs a pressing coverage pixel region centered on these coordinates. This region is typically represented by a fixed-radius circular structure or a discrete circle model based on pixel neighborhood expansion to simulate the contact influence range of the actual pressing tool on the reflective surface. After constructing the coverage region, the system determines the boundary range of the region based on the pixel distribution in the image coordinate system and applies spatial constraints to the pixels within the boundary to avoid invalid calculations caused by exceeding the effective detection area or the outer area of ​​the workpiece. Subsequently, the system performs point-by-point traversal processing on all pixels within the coverage region, extracting the coordinate information of unit pixels that meet the boundary conditions and forming a pixel set. Based on this, the system combines camera calibration parameters and the mapping relationship between pixels and actual physical dimensions to calculate the corresponding actual pixel area value for each unit pixel, for example, by converting it using a pixel size scaling factor or a perspective-corrected scale factor, thereby obtaining an area expression under a unified physical quantification standard. Finally, the system accumulates the actual areas of all unit pixels and outputs the cumulative coverage area value corresponding to the current pressing action. This calculation process not only considers the contribution of individual pixels, but also comprehensively reflects the spatial distribution of the entire pressed coverage area.

[0066] The above steps enable precise quantitative calculation of the area covered by the pressure, improving the accuracy and consistency of coverage area assessment.

[0067] Furthermore, the steps of obtaining the actual pixel area corresponding to each unit pixel point based on the image calibration parameters and accumulating the actual pixel areas specifically include: Obtain camera calibration parameters, including pixel size conversion coefficients, distortion correction parameters, and imaging plane coordinate mapping relationships; Distortion correction is performed on the pressed and covered pixel area based on camera calibration parameters, and the pixel coordinates are mapped to a unified imaging plane coordinate system; Based on the pixel size conversion factor, each unit pixel is converted into its corresponding actual physical area value, and the area weight of each unit pixel is calculated. The actual physical area values ​​corresponding to all unit pixels are summed to obtain the actual coverage area corresponding to the current pressing action.

[0068] In this embodiment, after the system acquires the camera calibration parameters, it first performs a structured analysis of the parameter system. The pixel size conversion coefficient is used to establish the proportional relationship between pixels and actual physical dimensions. Distortion correction parameters are used to correct radial and tangential distortions generated during lens imaging. The imaging plane coordinate mapping relationship describes the geometric transformation relationship between the pixel coordinate system and the unified workpiece plane coordinate system. After loading the parameters, the system performs distortion correction processing on the pressed-covered pixel area. By performing a reverse mapping operation on the original image pixels, the spatial offset caused by lens distortion is corrected, restoring the pixel distribution to an approximately ideal imaging plane structure. Subsequently, based on the coordinate mapping model constructed from the calibration results, the system uniformly transforms the corrected pixel coordinates to the standard imaging plane coordinate system, thereby ensuring the comparability of pixels in different regions at the same spatial scale. On this basis, the system performs scale conversion on each unit pixel according to the pixel size conversion coefficient, converting it into the corresponding actual physical area value, such as an area representation at the millimeter-square level. Furthermore, an area weighting factor is introduced to correct minor errors in edge regions caused by sampling density or perspective effects. Finally, the system sums up the actual physical areas of all unit pixels to obtain the coverage area value of the current pressing action in real space. This process realizes accurate mapping and unified quantitative expression from the pixel domain to the physical domain.

[0069] Through the above steps, a high-precision conversion from pixel area to actual physical area can be achieved, improving the spatial consistency and calibration accuracy of coverage area calculation.

[0070] In one specific embodiment of this application, a closed-loop control logic based on "real-time detection—instant alarm—on-site correction—re-inspection and release—full traceability" is implemented to address non-conforming products during processing. When the system determines, through AI visual detection, that there are defects such as underpressure, false pressure, or incomplete pressure, it immediately triggers an audible and visual alarm at the workstation and highlights the corresponding defect type and location on the interactive interface. Simultaneously, the quality judgment result is recorded to the edge control unit and the data management system. At this time, the production line adopts a non-automatic rejection or diversion strategy, that is, the product is not automatically removed, but the release permission is suspended through the equipment interlocking mechanism to prevent non-conforming products from entering the next process. The operator performs repressurization or re-pressing operations on the product at the original workstation according to the system prompts. The system performs real-time video detection and motion recognition again on the repressurization process, recalculates indicators such as displacement characteristics, dwell time, and coverage ratio, and only after the process requirements are met can the alarm state be lifted and the release permission be restored. For products that pass re-inspection, the system automatically records the number of repressurizations, the type of abnormality, and the handling results, and stores them in association with the original test data to form a complete quality traceability link, thereby realizing on-site closed-loop correction of non-conforming products and full-process data management.

[0071] In this embodiment, the defect detection method operates on an electronic device (e.g., Figure 1 The server shown can receive instructions or acquire data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future wireless connection methods.

[0072] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0073] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0074] Further reference Figure 4 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a defect detection device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0075] like Figure 4 As shown, the defect detection device 400 described in this embodiment includes: The video stream processing module 401 is used to acquire a continuous temporal video stream, extract the tool bounding box in the temporal video stream, and extract the vertical displacement sequence based on the trajectory of the center point of the tool bounding box to obtain the first displacement feature. The minimum point module 402 is used to extract the minimum point of the first displacement feature. If the displacement change rate between adjacent frames before and after the minimum point is lower than the preset change threshold, the corresponding number of continuous frames is extracted to determine the dwell time. The low-point pressing module 403 is used to extract the spatial location corresponding to the minimum point if the dwell time is greater than a preset time threshold, and obtain the target low-point pressing point. The coordinate mapping module 404 is used to extract the pixel position of the target pressing low point in the video frame, and obtain the real landing point coordinates by mapping the pixel position through the coordinate transformation matrix. The region identification module 405 is used to obtain a pre-established preset region and calculate the inclusion relationship between the actual landing point coordinates and the preset region. The area coverage module 406 is used to accumulate the corresponding unit pixel area to obtain the coverage area and calculate the coverage ratio if the actual landing point coordinates are located within the preset area. The leakage pressure defect module 407 is used to determine the existence of leakage pressure defect if the coverage ratio is less than a preset ratio threshold, and generate a first quality judgment result. The defect identification module 408 is used to classify the first quality judgment result and the vertical displacement sequence to determine the defect identification category.

[0076] Furthermore, the video stream processing module 401 specifically includes: The video acquisition submodule is used to continuously acquire video of the pressing operation process using an industrial camera installed above the workstation, and obtain a time-series video stream containing the movement of the pressing tool. The target detection submodule is used to perform target detection on each frame of the time-series video stream, identify the bounding box of the tool or the hand that is pressing, and extract the coordinates of the center point of the bounding box. The correlation tracking submodule is used to correlate and track the coordinates of the center point in each frame in chronological order, form a sequence of tool center point trajectories, and extract the vertical coordinate changes in the center point trajectory sequence. The time series analysis submodule is used to construct a vertical displacement sequence based on the vertical coordinate change, perform time series analysis on the vertical displacement sequence, extract the compression depth, displacement change trend curve and dwell characteristics, and obtain the first displacement feature.

[0077] Furthermore, the minimum point module 402 specifically includes: The temporal traversal submodule is used to perform temporal traversal on the vertical displacement sequence corresponding to the first displacement feature and extract local minimum points where the displacement value is lower than the adjacent temporal points as candidate low pressing points. The displacement change calculation submodule is used to calculate the displacement change rate between adjacent frames before and after the candidate press low point, and compare the displacement change rate with the preset change threshold. The pause determination submodule is used to determine that the pressing action corresponding to the candidate press low point is in a stable pause state if the displacement change rate before and after the candidate press low point is lower than the preset change threshold. The dwell time acquisition submodule is used to count the number of consecutive video frames corresponding to a stable dwell state, and to calculate the dwell time corresponding to the candidate low point of the press by combining the video frame rate.

[0078] Furthermore, the region identification module 405 specifically includes: The area division submodule is used to pre-establish the corresponding pressing standard area according to the current product model, and divide the pressing standard area into the central main pressing area, the edge pressing area and the four corner key pressing areas, and obtain the boundary coordinate information of each preset area; The actual landing point reading submodule is used to read the actual landing point coordinates and map the actual landing point coordinates to the corresponding area coordinate system; The region inclusion submodule is used to calculate whether the coordinates of the actual landing point are within the corresponding preset region boundary based on the spatial positional relationship between the point and the region, and to obtain the region inclusion relationship result. The region boundary range is determined based on the boundary coordinate information corresponding to each preset region. The area coverage marking submodule is used to mark the corresponding area as a valid coverage area if the actual landing point coordinates are within the preset area boundary range, and to record the corresponding area type and coverage count.

[0079] Further, please refer to Figure 5 The area coverage module 406 specifically includes: The pixel region establishment submodule 501 is used to establish the corresponding press coverage pixel region with the real landing point coordinates as the center when the real landing point coordinates are located within the preset area. The pixel area accumulation submodule 502 is used to extract the unit pixels in the pressed and covered pixel area, and accumulate the pixel area corresponding to each unit pixel to obtain the coverage area corresponding to the current pressing action. The coverage area overlay submodule 503 is used to overlay and count the coverage area corresponding to continuous pressing actions, and to remove duplicate pixels that are repeatedly covered, so as to obtain the cumulative coverage area of ​​the target area. The coverage ratio submodule 504 is used to obtain the total area corresponding to the preset area and calculate the coverage ratio based on the cumulative coverage area and the total area. The coverage ratio comparison submodule 505 is used to compare the coverage ratio with the preset coverage standard and output the corresponding area coverage result.

[0080] Furthermore, the pixel area accumulation submodule 502 specifically includes: The boundary range determination unit is used to generate the corresponding press coverage pixel area based on the actual landing point coordinates and the preset press radius, and to determine the boundary range corresponding to the press coverage pixel area. The pixel information unit is used to traverse each pixel within the pressed and covered pixel area and extract the coordinate information of the unit pixel within the boundary range; The pixel area accumulation unit is used to obtain the actual pixel area corresponding to each unit pixel point according to the image calibration parameters, and to accumulate and calculate the actual pixel area. The coverage area output unit is used to output the cumulative pixel area corresponding to the current pressing action as the coverage area corresponding to the current pressing action.

[0081] Furthermore, the pixel-area accumulation unit specifically includes: The camera calibration subunit is used to acquire camera calibration parameters, which include pixel size conversion coefficients, distortion correction parameters, and imaging plane coordinate mapping relationships. The distortion correction subunit is used to perform distortion correction on the pressed and covered pixel area based on camera calibration parameters and to map the pixel coordinates to a unified imaging plane coordinate system. The physical area conversion subunit is used to convert each unit pixel into the corresponding actual physical area value according to the pixel size conversion coefficient, and to calculate the area weight of each unit pixel. The area accumulation and summation subunit is used to accumulate and sum the actual physical area values ​​corresponding to all unit pixels to obtain the actual coverage area corresponding to the current pressing action.

[0082] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0083] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0084] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0085] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for defect detection methods. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0086] In some embodiments, the processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or to process data, for example, to execute computer-readable instructions for the defect detection method.

[0087] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0088] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the defect detection method described above.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0090] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A defect detection method, characterized in that, include: A continuous temporal video stream is acquired, a tool bounding box is extracted from the temporal video stream, and a vertical displacement sequence is extracted based on the trajectory of the center point of the tool bounding box to obtain the first displacement feature; Extract the minimum point of the first displacement feature. If the displacement change rate between adjacent frames before and after the minimum point is lower than a preset change threshold, extract the corresponding number of continuous frames to determine the dwell time. If the dwell time is greater than a preset time threshold, the spatial location corresponding to the minimum point is extracted to obtain the target low point of pressure. Extract the pixel position of the target press low point in the video frame, and obtain the real landing point coordinates by mapping the pixel position through a coordinate transformation matrix; Obtain a pre-established preset region and calculate the inclusion relationship between the actual landing point coordinates and the preset region; If the actual landing point coordinates are located within the preset area, the corresponding unit pixel area is accumulated to obtain the coverage area, and the coverage ratio is calculated. If the coverage ratio is less than a preset ratio threshold, a leakage pressure defect is determined, and a first quality judgment result is generated. The first quality judgment result and the vertical displacement sequence are classified to determine the defect identification category.

2. The defect detection method as described in claim 1, characterized in that, The steps of acquiring a continuous temporal video stream, extracting tool bounding boxes from the temporal video stream, and extracting a vertical displacement sequence based on the trajectory of the center point of the tool bounding box to obtain a first displacement feature specifically include: The pressing operation process is continuously captured by an industrial camera installed above the workstation, resulting in a time-series video stream containing the movement of the pressing tool. Target detection is performed on each frame of the time-series video stream to identify the bounding box of the pressing tool or the pressing hand, and the center point coordinates of the bounding box are extracted. The coordinates of the center points in each frame are correlated and tracked in chronological order to form a sequence of tool center point trajectories, and the vertical coordinate changes in the center point trajectory sequence are extracted. Based on the vertical coordinate change, a vertical displacement sequence is constructed, and a time series analysis is performed on the vertical displacement sequence to extract the compression depth, displacement change trend curve, and dwell characteristics, thereby obtaining the first displacement feature.

3. The defect detection method as described in claim 2, characterized in that, The step of extracting the minimum point of the first displacement feature, and if the displacement change rate between adjacent frames before and after the minimum point is lower than a preset change threshold, then extracting the corresponding number of continuous frames to determine the dwell time, specifically includes: The vertical displacement sequence corresponding to the first displacement feature is traversed in time sequence, and local minimum points where the displacement value is lower than the adjacent time sequence points are extracted as candidate low pressing points. Calculate the displacement change rate between adjacent frames before and after the candidate press low point, and compare the displacement change rate with a preset change threshold. If the displacement change rate before and after the candidate low point is lower than the preset change threshold, then the pressing action corresponding to the candidate low point is determined to be in a stable dwell state. The number of consecutive video frames corresponding to the stable dwell state is counted, and the dwell time corresponding to the candidate low press point is obtained by combining the video frame rate.

4. The defect detection method as described in claim 1, characterized in that, The steps of obtaining a pre-established preset region and calculating the inclusion relationship between the actual landing point coordinates and the preset region specifically include: Based on the current product model, a corresponding standard pressing area is pre-established, and the standard pressing area is divided into a central main pressing area, an edge pressing area, and four corner key pressing areas, and the boundary coordinate information corresponding to each pre-established area is obtained. Read the actual landing point coordinates and map the actual landing point coordinates to the corresponding regional coordinate system; Based on the spatial relationship between points and regions, the coordinates of the actual landing point are calculated to determine whether they are within the boundary range of the corresponding preset region, thereby obtaining the region inclusion relationship result. The boundary range of the region is determined based on the boundary coordinate information corresponding to each preset region. If the actual landing point coordinates are within the boundary range of the preset area, the corresponding area is marked as an effective coverage area, and the corresponding area type and coverage count are recorded.

5. The defect detection method as described in claim 1, characterized in that, The step of accumulating the corresponding unit pixel area to obtain the coverage area and calculating the coverage ratio if the actual landing point coordinates are located within the preset area specifically includes: When the actual landing point coordinates are located within the preset area, a corresponding press coverage pixel area is established with the actual landing point coordinates as the center; Extract the unit pixels in the pressed and covered pixel area, and accumulate the pixel areas corresponding to each unit pixel to obtain the coverage area corresponding to the current pressing action; The coverage area corresponding to continuous pressing actions is superimposed and statistically analyzed, and duplicate pixel areas are removed to obtain the cumulative coverage area of ​​the target area. Obtain the total area corresponding to the preset area, and calculate the coverage ratio based on the cumulative coverage area and the total area. The coverage ratio is compared with a preset coverage standard, and the corresponding regional coverage result is output.

6. The defect detection method as described in claim 5, characterized in that, The step of extracting unit pixels in the pressed and covered pixel region and accumulating the pixel areas corresponding to each unit pixel to obtain the coverage area corresponding to the current pressing action specifically includes: Based on the actual landing point coordinates and the preset pressing radius, a corresponding pressing coverage pixel area is generated, and the boundary range corresponding to the pressing coverage pixel area is determined. Traverse each pixel within the pressed and covered pixel area, and extract the coordinate information of the unit pixel within the boundary range; The actual pixel area corresponding to each unit pixel is obtained according to the image calibration parameters, and the actual pixel area is accumulated and calculated. Output the cumulative pixel area corresponding to the current press action as the coverage area corresponding to the current press action.

7. The defect detection method as described in claim 6, characterized in that, The step of obtaining the actual pixel area corresponding to each unit pixel point according to the image calibration parameters, and accumulating the actual pixel areas, specifically includes: Obtain camera calibration parameters, wherein the camera calibration parameters include pixel size conversion coefficients, distortion correction parameters, and imaging plane coordinate mapping relationships; Based on the camera calibration parameters, distortion correction is performed on the pressed and covered pixel area, and the pixel coordinates are mapped to a unified imaging plane coordinate system; Based on the pixel size conversion coefficient, each unit pixel is converted into its corresponding actual physical area value, and the area weight of each unit pixel is calculated. The actual physical area values ​​corresponding to all unit pixels are summed to obtain the actual coverage area corresponding to the current pressing action.

8. A defect detection device, characterized in that, include: The video stream processing module is used to acquire a continuous temporal video stream, extract the tool bounding box in the temporal video stream, and extract the vertical displacement sequence based on the trajectory of the center point of the tool bounding box to obtain the first displacement feature. The minimum point module is used to extract the minimum point of the first displacement feature. If the displacement change rate between adjacent frames before and after the minimum point is lower than a preset change threshold, the corresponding number of continuous frames is extracted to determine the dwell time. The low-point pressing module is used to extract the spatial location corresponding to the minimum point and obtain the target low-point pressing if the dwell time is greater than a preset time threshold. The coordinate mapping module is used to extract the pixel position of the target press low point in the video frame, and to obtain the real landing point coordinates by mapping the pixel position through a coordinate transformation matrix. The region identification module is used to acquire a pre-established preset region and calculate the inclusion relationship between the actual landing point coordinates and the preset region. The region coverage module is used to accumulate the corresponding unit pixel area to obtain the coverage area and calculate the coverage ratio if the actual landing point coordinates are located within the preset region. The leakage pressure defect module is used to determine the existence of leakage pressure defect and generate a first quality judgment result if the coverage ratio is less than a preset ratio threshold. The defect identification module is used to classify the first quality judgment result and the vertical displacement sequence to determine the defect identification category.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the defect detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the defect detection method as described in any one of claims 1 to 7.

Images