A Deep Learning-Based Method and System for Testing the Front Panel of a Multimedia Terminal

Abstract

This invention discloses an automated testing method and system for the front panel of a multimedia terminal based on deep learning, relating to the fields of machine vision and industrial inspection technology. It includes: issuing preset display commands to the multimedia terminal under test via an industrial control terminal; acquiring front panel images after the terminal completes its display response; and locating small targets such as digital tubes and LEDs based on an improved target detection model. Combining region cropping and content recognition models, it identifies the displayed digits on the digital tubes and the colors and on / off states of the LEDs, generating structured recognition results based on preset state judgment rules. A multi-frame voting fusion mechanism improves detection stability, and a re-shooting and anomaly handling process is triggered based on recognition confidence. Abnormal samples are incorporated into the incremental training system, and model versions are tracked and managed to achieve continuous optimization of the detection model. The detection results are linked with the production execution system to achieve centralized storage and quality traceability of the detection data.

Description

Technical Field

[0001] This invention relates to the fields of machine vision and industrial inspection technology, specifically to a method and system for testing the front panel of a multimedia terminal based on deep learning. Background Technology

[0002] With the rapid development of intelligent manufacturing and the consumer electronics industry, the output of electronic products is enormous, and the requirements for their production quality and functional reliability are increasing daily. The front panel of a multimedia terminal typically includes various components such as power indicator lights, function buttons, and digital display tubes. Their assembly integrity, appearance quality, and functional status (such as indicator light color and on / off state) are critical items that must be inspected before shipment. Traditional inspection methods relying on manual visual inspection suffer from low efficiency, high cost, strong subjectivity, and susceptibility to fatigue-induced omissions, and can no longer meet the demands of modern, high-paced, and highly consistent production.

[0003] Currently, automated detection technologies for this type of problem are mainly divided into two categories: traditional machine vision solutions and deep learning (DL)-based target detection solutions. Traditional machine vision solutions are extremely sensitive to changes in lighting, slight product deformation, protective films, and background interference, requiring careful design of light sources and lighting environments, and have weak generalization ability. They are difficult to effectively detect state recognition (such as distinguishing whether an indicator light is "on" or "off," especially when it is faintly lit) or minor defects (such as tiny scratches or printing flaws). Each new product or defect requires engineers to redesign and debug algorithm parameters, which is time-consuming and labor-intensive. Deep learning-based target detection solutions are not effective at detecting small targets: indicators, character markings, and other targets on the front panel of multimedia terminals are extremely small and occupy a small proportion of the entire image. The deep networks of standard models are prone to losing feature information of small targets, leading to missed detections or inaccurate localization. General detection models usually only output "category" and "location," and cannot directly output "state." Standard models involve large amounts of computation, making it difficult to meet the high real-time requirements of production lines while ensuring accuracy; moreover, they require a large amount of training data, and in actual production, defect samples are scarce, resulting in a low detection rate of rare defects by the model.

[0004] While existing technologies include device display status recognition solutions based on single models, they mostly focus on general scenarios and are not optimized for the layout characteristics of digital tubes and LED lights on the front panel of multimedia terminals. Furthermore, they do not achieve full-process linkage with production control systems and MES systems, thus failing to meet the testing needs of large-scale production of multimedia terminals. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that the existing device display status recognition schemes are not optimized for the layout features of digital tubes and LED lights on the front panel of multimedia terminals, resulting in inaccurate defect detection. The purpose is to provide a multimedia terminal front panel testing method and system based on deep learning, which solves how to improve detection accuracy through step-by-step recognition based on deep learning and realize real-time linkage between recognition results and MES system.

[0006] This invention is achieved through the following technical solution:

[0007] A deep learning-based method for testing the front panel of a multimedia terminal includes the following steps:

[0008] The industrial control terminal sends a preset display command to the multimedia terminal under test. The display command is used to control the digital tube to display the target character sequence and control the LED lights to display the target color state. After the multimedia terminal under test completes the display response, the image acquisition module acquires the image of its front panel.

[0009] The front panel image is input into an improved target detection model for processing, and the target position coordinates corresponding to the digital tube area and the LED light area are output. The front panel image is cropped based on the target position coordinates to obtain the corresponding digital tube sub-image and LED light image.

[0010] The digital tube image is input into a digital recognition model for recognition to obtain a digital recognition result containing the displayed digital value and corresponding confidence parameters; the LED light image is input into a color recognition model for recognition to obtain a color recognition result containing the color category and corresponding confidence parameters; based on preset state determination rules, the color recognition result and brightness feature parameters are jointly analyzed to determine the on / off state and color category of the LED light, thus obtaining a color state recognition result; the brightness feature parameters are extracted based on the LED light image;

[0011] The digital recognition result and color status recognition result are compared with the corresponding preset display instructions to generate a detection judgment result. When the detection judgment result meets the preset consistency conditions, the multimedia terminal under test is judged to be qualified. When it does not meet the conditions, an abnormal handling process is executed. The abnormal handling process includes at least triggering a re-scanning and re-recognition or an abnormal sample storage operation.

[0012] The detection and judgment results and the corresponding detection process data are uploaded to the production execution system.

[0013] Preferably, the improved target detection model includes at least one layer of small target-specific detection head and a high-resolution feature fusion layer to enhance the ability to extract features of small targets such as digital tube characters and LED lights.

[0014] Preferably, before inputting the front panel image into the improved target detection model for processing, the method further includes: performing region localization on the front panel image based on the front panel structural layout features, and establishing candidate region constraints for the digital tube and LED lights; the target detection model performs target detection only within the candidate region constraints.

[0015] Preferably, cropping the front panel image based on the target location coordinates includes: performing boundary expansion and smoothing processing on the target location coordinates, and performing stability correction on the cropped area based on the continuous frame detection results; and performing consistency verification on the recognition results based on the effective pixel ratio of the cropped area.

[0016] Preferably, after acquiring the front panel image through the image acquisition module, image acquisition, target localization, region cropping and recognition processing are performed on the K consecutively acquired front panel images to obtain the corresponding K sets of recognition results, where K is a preset positive integer;

[0017] The digit recognition results and color state recognition results in the K groups of recognition results are fused by voting, and the final detection judgment result is determined based on the majority consensus principle or the confidence weighting principle.

[0018] Preferably, the state determination rule includes:

[0019] Perform color space conversion and brightness normalization on the LED light image;

[0020] Extract the percentage of bright pixels, average brightness value, and saturation parameters;

[0021] Based on the relationship between the parameters and the preset threshold range, the on / off state and color category of the LED light are determined.

[0022] Preferably, when the confidence level of the recognition result is lower than the preset confidence threshold, or when there is a conflict in the recognition results of consecutive frames, the re-shooting and re-recognition process is triggered, and the detection judgment result is updated based on the re-shooting result.

[0023] Preferably, when performing anomaly handling, the corresponding original image, cropped sub-image, recognition result, instruction parameters, and time information are stored in the anomaly sample pool;

[0024] Periodic incremental training is performed based on sample data in the abnormal sample pool to update the parameters of the target detection model and the recognition model.

[0025] Preferably, when uploading the detection judgment result and the corresponding detection process data to the production execution system, a corresponding model version identifier is assigned for each model update; and the model version identifier is associated with the detection object, detection result and time information and stored in the production execution system.

[0026] This invention also provides a deep learning-based multimedia terminal front panel testing system for implementing the aforementioned deep learning-based multimedia terminal front panel testing method, comprising:

[0027] An industrial control terminal is used to send preset display commands to the multimedia terminal under test via a communication interface to control the digital tube to display the target character sequence and control the LED lights to display the target color status.

[0028] An image acquisition device, including an industrial camera, is used to acquire an image of the front panel of a multimedia terminal under test after it has completed a display response.

[0029] A computing processing device includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used for:

[0030] Target localization is performed on the front panel image, and the target position coordinates of the digital tube area and the LED light area are output; the area is cropped based on the target position coordinates to obtain digital tube sub-images and LED light images respectively; digital recognition is performed on the digital tube images and the digital recognition result containing the displayed digital value and the corresponding confidence parameter is output;

[0031] The LED light sub-image is color-recognized, and the on / off state and color category of the LED light are determined based on a preset state determination rule. The color recognition result containing the on / off state, color category and corresponding confidence parameters is output.

[0032] The consistency of the digital recognition result, the color state recognition result and the preset display instruction is compared to generate a detection judgment result, and a re-shooting process is performed when the recognition confidence is lower than the preset confidence threshold or when the recognition results of multiple frames are inconsistent.

[0033] The production execution system is used to receive and store the detection judgment results and detection process data, and to store abnormal sample data and record the model version identifier when an abnormality is detected.

[0034] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0035] This invention constructs a dual-model collaborative recognition mechanism based on an improved target detection model and a content recognition model, and combines a small target detection head, a high-resolution feature fusion layer, and front panel layout constraints to effectively improve the positioning accuracy and detection stability of small targets such as digital tube characters and LED lights in complex lighting environments, and reduce the false detection rate and missed detection rate.

[0036] This invention constructs a joint judgment rule for on / off status and color based on color space conversion, brightness normalization, and bright area pixel ratio analysis, thereby achieving refined identification of the on / off status and color category of LED lights. This solves the problem that existing general detection models cannot directly output status information and improves the reliability of display status detection.

[0037] This invention introduces a multi-frame acquisition and voting fusion mechanism to perform consistency analysis and fusion judgment on the recognition results of consecutive frames, effectively suppressing misjudgment caused by reflection, jitter and instantaneous noise, and improving the robustness and stability of the detection results.

[0038] This invention constructs a low-confidence re-shooting and anomaly handling mechanism based on identification confidence and inter-frame consistency, realizing automatic screening and verification of uncertain samples, reducing the frequency of manual re-inspection, and improving the detection efficiency of the production line.

[0039] This invention enables the detection model to be continuously optimized and iterated by establishing an abnormal sample pool, periodic incremental training, and a model version traceability mechanism, and realizes the correlation management between detection results and model versions, thereby improving the system's long-term detection capability for rare defects and the level of quality traceability.

[0040] This invention deeply integrates the automated testing process with the production execution system, enabling centralized storage and real-time analysis of testing data. This facilitates quality control and anomaly tracing during the production process and improves the overall intelligence level of testing in the large-scale production of multimedia terminals. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0042] Figure 1 This is a structural diagram of a multimedia terminal front panel testing system based on deep learning. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of this invention are only for explaining this invention and are not intended to limit this invention.

[0044] Example 1

[0045] A deep learning-based method for testing the front panel of a multimedia terminal includes the following steps:

[0046] The industrial control terminal sends a preset display command to the multimedia terminal under test. The display command is used to control the digital tube to display the target character sequence and control the LED lights to display the target color state. After the multimedia terminal under test completes the display response, the image acquisition module acquires the image of its front panel.

[0047] The front panel image is input into an improved target detection model for processing, and the target position coordinates corresponding to the digital tube area and the LED light area are output. The front panel image is cropped based on the target position coordinates to obtain the corresponding digital tube sub-image and LED light image.

[0048] The digital tube image is input into a digital recognition model for recognition to obtain a digital recognition result containing the displayed digital value and corresponding confidence parameters; the LED light image is input into a color recognition model for recognition to obtain a color recognition result containing the color category and corresponding confidence parameters; based on preset state determination rules, the color recognition result and brightness feature parameters are jointly analyzed to determine the on / off state and color category of the LED light, thus obtaining a color state recognition result; the brightness feature parameters are extracted based on the LED light image;

[0049] The digital recognition result and color status recognition result are compared with the corresponding preset display instructions to generate a detection judgment result. When the detection judgment result meets the preset consistency conditions, the multimedia terminal under test is judged to be qualified. When it does not meet the conditions, an abnormal handling process is executed. The abnormal handling process includes at least triggering a re-scanning and re-recognition or an abnormal sample storage operation.

[0050] The detection and judgment results and the corresponding detection process data are uploaded to the production execution system.

[0051] In this embodiment, the digit recognition result is a structured data object, which includes at least: the digit value num displayed on the digital tube and the corresponding confidence parameter conf_num; optionally, it also includes the recognition frame number frame_id, the coordinates bbox_num of the digital tube cropping area, and the timestamp t.

[0052] The color state recognition result is a structured data object, including at least: color category (color), on / off state (on_off), and corresponding confidence parameter (conf_led); optionally, it also includes the coordinates of the LED light clipping region (bbox_led), the brightness feature parameter set (Lp), and the timestamp (t). The aforementioned confidence parameter can be the class probability, confidence score, or calibrated confidence score output by the model, and its value range can be [0,1] or other equivalent intervals.

[0053] In this embodiment, the brightness feature parameters are obtained by extracting from the LED lamp sub-image. Exemplarily, first, the color space of the LED lamp sub-image is converted (for example, converted to the HSV or Lab space) and brightness normalization processing is performed; subsequently, parameters such as the proportion of bright area pixels r_high, the average brightness value μ_v, and the saturation statistic s_v are extracted. Among them, the proportion of bright area pixels r_high can be defined as the ratio of the number of pixels with a brightness component greater than the preset brightness threshold Th_v in the LED lamp sub-image to the total number of pixels.

[0054] The on / off state is determined based on the relationship between the brightness feature parameters and the preset threshold interval: it is determined as "on" when r_high≥Th_on, and determined as "off" when r_high≤Th_off, where Th_on and Th_off are preset thresholds and satisfy Th_on>Th_off; when Th_off<r_high<Th_on, the recognition result of this frame can be marked as uncertain and enter the retake and re-recognition process. The color category color can be output by the color recognition model and can be combined with the on / off determination result for consistency verification. For example, when it is determined as "off", the color category output is invalid or set to "none".

[0055] In this embodiment, the improved object detection model includes at least one small target dedicated detection head and a high-resolution feature fusion layer to enhance the ability to extract the features of the digital tube characters and the LED lamp small targets. Exemplarily, the model sets a small target dedicated detection head and a high-resolution feature fusion layer in the feature extraction network to enhance the ability to retain small target features; the feature fusion layer can be implemented by means of upsampling and feature splicing, feature addition, or a combination thereof.

[0056] Furthermore, since the layout of the digital tube and the LED lamp on the front panel of the multimedia terminal is relatively fixed, the area of the front panel image can be located before object detection and a candidate area constraint range can be established. The object detection model only performs inference within this candidate area to reduce false detection caused by background interference and improve the inference efficiency. The candidate area constraint range can be determined based on fixture fixed positioning, front panel outer contour / corner positioning, or positioning hole matching, etc.

[0057] In this embodiment, before inputting the front panel image into the improved object detection model for processing, it further includes: based on the front panel structure layout features, performing area positioning on the front panel image and establishing a candidate area constraint range for the digital tube and the LED lamp; the object detection model only performs object detection within the candidate area constraint range.

[0058] In this embodiment, cropping the front panel image based on the target location coordinates includes: performing boundary expansion and smoothing processing on the target location coordinates, and performing stability correction on the cropped area based on the continuous frame detection results; and performing consistency verification on the recognition results based on the effective pixel ratio of the cropped area.

[0059] Specifically, when cropping the front panel image based on the target position coordinates, the target position coordinates can be extended to cover the light diffusion area, and the target position coordinates of consecutive frames can be smoothed to suppress the effect of jitter. For example, the bounding box can be padded with a preset ratio, and the bounding box of consecutive frames can be stabilized by using a moving average or equivalent filtering method.

[0060] In addition, consistency verification of the recognition results can be performed based on the effective pixel ratio of the cropped area. For example, when the effective bright area pixel ratio in the LED light sub-image is lower than the preset threshold Th_eff, the color recognition result of the frame is considered unreliable and a reshoot or repositioning is triggered.

[0061] In this embodiment, after the front panel image is acquired by the image acquisition module, image acquisition, target localization, region cropping and recognition processing are performed on the K consecutively acquired front panel images to obtain K sets of corresponding recognition results, where K is a preset positive integer.

[0062] The digit recognition results and color state recognition results in the K groups of recognition results are fused by voting, and the final detection judgment result is determined based on the majority consensus principle or the confidence weighting principle.

[0063] Specifically, for K consecutively acquired front panel images, target localization, region cropping and recognition processing are performed on each frame to obtain K sets of recognition results, where K is a preset positive integer (e.g., K=3).

[0064] When performing voting fusion on the K groups of recognition results, the voting objects include at least the displayed numerical value `num`, the color category `color`, and the on / off state `on_off`. For example, a majority consensus principle can be used: when a candidate result appears a certain number of times in K frames reaches a preset consensus threshold (e.g., not less than ⌈K / 2⌉), the candidate result is determined as the final recognition result; or a confidence-weighted principle can be used: the candidate results in each frame are weighted and accumulated according to their confidence level `conf`, and the one with the highest weighted score is selected as the final recognition result. If the voting results still cannot meet the preset consensus condition, an exception handling process is triggered.

[0065] In this embodiment, the state determination rules include:

[0066] Perform color space conversion and brightness normalization on the LED light image;

[0067] Extract the percentage of bright pixels, average brightness value, and saturation parameters;

[0068] Based on the relationship between the parameters and the preset threshold range, the on / off state and color category of the LED light are determined.

[0069] In this embodiment, when the confidence level of the recognition result is lower than the preset confidence threshold, or when there is a conflict in the recognition results of consecutive frames, the re-shooting and re-recognition process is triggered, and the detection judgment result is updated based on the re-shooting result.

[0070] In this embodiment, when performing exception handling, the corresponding original image, cropped sub-image, recognition result, instruction parameters and time information are stored in the exception sample pool;

[0071] Periodic incremental training is performed based on sample data in the abnormal sample pool to update the parameters of the target detection model and the recognition model.

[0072] When the recognition confidence level is lower than the preset confidence threshold, or when there are conflicts in the recognition results of multiple frames, or when the recognition results are inconsistent with the preset display instructions, the corresponding original image, cropped sub-image, recognition result, instruction parameters, frame number, time information, workstation and batch information are stored in the abnormal sample pool for periodic incremental training to update the parameters of the target detection model and recognition model. During model updates, a model version identifier (version_id) is assigned to each update, and the version_id is associated with the detection object, detection result, and time information and stored in the production execution system to achieve model version traceability of the detection results.

[0073] In this embodiment, when the detection judgment result and the corresponding detection process data are uploaded to the production execution system, a corresponding model version identifier is assigned for each model update; and the model version identifier is associated with the detection object, detection result and time information and stored in the production execution system.

[0074] In summary, this invention addresses the automated testing needs of small display components such as digital tubes and LED lights on the front panel of multimedia terminals by constructing a multi-stage collaborative recognition and closed-loop optimization testing method based on deep learning. It achieves command-driven image acquisition through collaboration between an industrial control terminal and an image acquisition module, and completes precise localization and state recognition of small targets based on improved target detection and content recognition models.

[0075] This invention combines front panel structural layout constraints, small target feature enhancement, multi-frame voting fusion, and a joint determination rule for brightness / off state and color to achieve stable detection of display status under complex lighting conditions, effectively suppressing the interference of reflections, jitter, and transient noise on the recognition results. Simultaneously, through recognition confidence assessment, anomaly handling, and anomaly sample pool management mechanisms, low-confidence and conflicting samples are automatically included in the incremental training process, enabling continuous optimization and iteration of the detection model.

[0076] Furthermore, by associating and storing the detection results with model version information within the production execution system, this invention constructs a complete quality traceability and analysis system, providing reliable data support for tracing the source of production process anomalies and quality control. Overall, this invention improves the system's robustness, adaptability, and long-term stable operation while ensuring detection accuracy and real-time performance, making it suitable for automated display detection applications in large-scale multimedia terminal production environments.

[0077] Example 2

[0078] A deep learning-based multimedia terminal front panel testing system is provided to implement the deep learning-based multimedia terminal front panel testing method of Embodiment 1, such as... Figure 1 As shown, the automated testing system of the present invention includes a production control module, an image acquisition device, a server-side recognition module, and a production execution system module. Wherein:

[0079] The production control module is an industrial control terminal, used to send preset display instructions to the multimedia terminal under test through a communication interface to control the digital tube to display the target character sequence and control the LED lights to display the target color status.

[0080] The image acquisition device is an industrial camera, used to acquire the image of the front panel of the multimedia terminal under test after it completes the display response.

[0081] The server-side recognition module is deployed in a server or industrial control computer. It has a built-in target detection model and content recognition model, which are used to locate targets, crop regions, recognize numbers and color status in front panel images, and generate detection and judgment results.

[0082] The server-side identification module is a computing processing device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, is used for:

[0083] Target localization is performed on the front panel image, and the target position coordinates of the digital tube area and the LED light area are output; the area is cropped based on the target position coordinates to obtain digital tube sub-images and LED light images respectively; digital recognition is performed on the digital tube images and the digital recognition result containing the displayed digital value and the corresponding confidence parameter is output;

[0084] The LED light sub-image is color-recognized, and the on / off state and color category of the LED light are determined based on a preset state determination rule. The color recognition result containing the on / off state, color category and corresponding confidence parameters is output.

[0085] The consistency of the digital recognition result, the color state recognition result and the preset display instruction is compared to generate a detection judgment result, and a re-shooting process is performed when the recognition confidence is lower than the preset confidence threshold or when the recognition results of multiple frames are inconsistent.

[0086] The MES server is a production execution system used to receive and store the detection judgment results and detection process data, and to store abnormal sample data and record model version identifier when an abnormality is detected.

[0087] During the inspection process, the production control module establishes data connections with the image acquisition device, the server-side recognition module, and the production execution system module through industrial Ethernet, serial port, or other communication methods to realize the interactive transmission of instruction information, image data, and inspection results, thereby forming a complete automated inspection closed-loop system.

[0088] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0090] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0091] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Those skilled in the art will understand that all or part of the steps in the above facts and methods can be implemented by a program instructing related hardware. The program or the program described therein can be stored in a computer-readable storage medium. When the program is executed, it includes the following steps: at this time, the corresponding method steps are introduced. The storage medium can be ROM / RAM, magnetic disk, optical disk, etc.

[0093] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for testing the front panel of a multimedia terminal based on deep learning, characterized in that, Includes the following steps: The industrial control terminal sends a preset display command to the multimedia terminal under test. The display command is used to control the digital tube to display the target character sequence and control the LED lights to display the target color state. After the multimedia terminal under test completes the display response, the image acquisition module acquires the image of its front panel. The front panel image is input into an improved target detection model for processing, and the target position coordinates corresponding to the digital tube area and the LED light area are output. The front panel image is cropped based on the target position coordinates to obtain the corresponding digital tube sub-image and LED light image. The image of the digital tube is input into the digital recognition model for recognition, and a digital recognition result containing the displayed digital value and the corresponding confidence parameter is obtained; The LED light image is input into a color recognition model for recognition, and a color recognition result containing color category and corresponding confidence parameters is obtained. Based on a preset state determination rule, the color recognition result and brightness feature parameters are jointly analyzed to determine the on / off state and color category of the LED light, and a color state recognition result is obtained. The brightness feature parameters are extracted based on the LED light image. The digital recognition results and color state recognition results are compared with the corresponding preset display instructions to generate a detection judgment result. When the detection result meets the preset consistency condition, the multimedia terminal under test is deemed to be qualified; when it does not meet the condition, an exception handling process is executed, which includes at least triggering a re-encoding or an exception sample storage operation. The detection and judgment results and the corresponding detection process data are uploaded to the production execution system.

2. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, The improved target detection model includes at least one layer of small target-specific detection head and a high-resolution feature fusion layer.

3. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 2, characterized in that, Before inputting the front panel image into the improved target detection model for processing, the method further includes: performing region localization on the front panel image based on the front panel structural layout features, and establishing candidate region constraints for the digital tube and LED lights; the target detection model only performs target detection within the candidate region constraints.

4. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, Cropping the front panel image based on the target location coordinates includes: performing boundary expansion and smoothing processing on the target location coordinates, and performing stability correction on the cropped area based on the continuous frame detection results; and performing consistency verification on the recognition results based on the effective pixel ratio of the cropped area.

5. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, After acquiring the front panel image through the image acquisition module, image acquisition, target localization, region cropping and recognition processing are performed on the K consecutively acquired front panel images to obtain the corresponding K sets of recognition results, where K is a preset positive integer; The digit recognition results and color state recognition results in the K groups of recognition results are fused by voting, and the final detection judgment result is determined based on the majority consensus principle or the confidence weighting principle.

6. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, The state determination rules include: Perform color space conversion and brightness normalization on the LED light image; Extract the percentage of bright pixels, average brightness value, and saturation parameters; Based on the relationship between the parameters and the preset threshold range, the on / off state and color category of the LED light are determined.

7. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, When the confidence level of the recognition result is lower than the preset confidence threshold, or when there is a conflict between the recognition results of consecutive frames, the re-shooting and re-recognition process is triggered, and the detection judgment result is updated based on the re-shooting result.

8. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, When performing exception handling, the corresponding original image, cropped sub-image, recognition result, instruction parameters and time information are stored in the exception sample pool; Periodic incremental training is performed based on sample data in the abnormal sample pool to update the parameters of the target detection model and the recognition model.

9. The method for testing the front panel of a multimedia terminal based on deep learning according to claim 1, characterized in that, When the detection and judgment results and the corresponding detection process data are uploaded to the production execution system, a corresponding model version identifier is assigned for each model update; and the model version identifier is associated with the detection object, detection result and time information and stored in the production execution system.

10. A deep learning-based multimedia terminal front panel testing system, used to implement the deep learning-based multimedia terminal front panel testing method described in claims 1-9, characterized in that, include: An industrial control terminal is used to send preset display commands to the multimedia terminal under test via a communication interface to control the digital tube to display the target character sequence and control the LED lights to display the target color status. An image acquisition device, including an industrial camera, is used to acquire an image of the front panel of a multimedia terminal under test after it has completed a display response. A computing processing device includes a processor and a memory, wherein the memory stores a computer program, which, when executed by the processor, is used for: Target localization is performed on the front panel image, and the target position coordinates of the digital tube area and the LED light area are output; the area is cropped based on the target position coordinates to obtain digital tube sub-images and LED light images respectively; digital recognition is performed on the digital tube images and the digital recognition result containing the displayed digital value and the corresponding confidence parameter is output; The LED light sub-image is color-recognized, and the on / off state and color category of the LED light are determined based on a preset state determination rule. The color recognition result containing the on / off state, color category and corresponding confidence parameters is output. The consistency of the digital recognition result, the color state recognition result and the preset display instruction is compared to generate a detection judgment result, and a re-shooting process is performed when the recognition confidence is lower than the preset confidence threshold or when the recognition results of multiple frames are inconsistent. The production execution system is used to receive and store the detection judgment results and detection process data, and to store abnormal sample data and record the model version identifier when an abnormality is detected.

Images