Dashboard Final Inspection System and Method

Abstract

The present invention relates to an instrument panel final inspection system and method. The method includes: performing multi-level feature extraction on the acquired detection image of the instrument panel to be detected through an instrument panel feature extractor based on a deep neural network to obtain an instrument panel shallow feature map and an instrument panel semantic feature map; passing the instrument panel shallow feature map and the instrument panel semantic feature map through an instrument panel multi-scale feature enhancement expression module to obtain an instrument panel multi-scale shallow feature map and an instrument panel multi-scale semantic feature map; based on the instrument panel multi-scale semantic feature map, performing region transfer based on an attention mechanism on the instrument panel multi-scale shallow feature map to obtain a foreground prominent instrument panel shallow feature, so as to determine whether there are defects in the instrument panel to be detected. In this way, intelligent final inspection of the instrument panel can be realized, thereby avoiding the errors and low efficiency problems brought by the traditional manual final inspection method, being able to well handle complex instrument panel defect problems, and ensuring that the quality of the instrument panel product meets the standard requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of instrument panels, and specifically, to an instrument panel final inspection system and method. Background Art

[0002] The instrument panel is a crucial component in an automobile, capable of providing key information about the vehicle's status and performance. Final inspection of the instrument panel is an important link in the manufacturing industry, used to ensure that the produced instrument panel products meet quality standards and specification requirements.

[0003] In the traditional process of final inspection of the instrument panel, manual visual inspection is usually relied on to conduct quality inspection of the instrument panel. Manual visual inspection is easily affected by human subjective factors, and different operators may have different judgment criteria, resulting in inconsistencies in the inspection results. Moreover, manual visual inspection requires a large amount of human input, with low work efficiency, and long-term repetitive work is likely to cause operator fatigue, thereby affecting the inspection accuracy. In addition, the traditional instrument panel final inspection scheme usually needs to wait for the completion of manual inspection to obtain the results, and cannot provide real-time feedback on possible problems on the production line, affecting production efficiency and product quality. At the same time, there may be various types of defects in the instrument panel products, and traditional detection methods are difficult to effectively handle complex situations, prone to missed detections or misjudgments.

[0004] Therefore, an optimized instrument panel final inspection system is desired. Summary of the Invention

[0005] This Summary of the Invention section is provided to introduce concepts in a brief form, which will be described in detail in the subsequent Detailed Description section. This Summary of the Invention section is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0006] In a first aspect, the present invention provides an instrument panel final inspection system, the system comprising:

[0007] An instrument panel image acquisition module, configured to acquire a detection image of the instrument panel to be detected collected by a camera;

[0008] An instrument panel multi-level feature extraction module, configured to perform multi-level feature extraction on the detection image of the instrument panel to be detected through an instrument panel feature extractor based on a deep neural network to obtain an instrument panel shallow feature map and an instrument panel semantic feature map;

[0009] An instrument panel feature enhancement and expression module, configured to pass the instrument panel shallow feature map and the instrument panel semantic feature map through an instrument panel multi-scale feature enhancement and expression module to obtain an instrument panel multi-scale shallow feature map and an instrument panel multi-scale semantic feature map;

[0010] The dashboard semantic information transfer and foreground highlighting shallow feature module is used to perform region transfer based on the attention mechanism on the dashboard multi-scale shallow feature map based on the dashboard multi-scale semantic feature map to obtain the foreground highlighted dashboard shallow feature;

[0011] The dashboard defect detection module is used to determine whether there is a defect in the detected dashboard based on the foreground highlighted dashboard shallow feature.

[0012] Optionally, the dashboard feature extractor based on the deep neural network is a dashboard feature extractor based on the pyramid network.

[0013] Optionally, the dashboard feature enhancement and expression module includes: a first convolutional unit for passing the dashboard shallow feature map through a first convolutional layer with a 1×1 convolutional kernel in the dashboard multi-scale feature enhancement and expression module to obtain a channel-transformed dashboard shallow feature map; a feature map splitting unit for splitting the channel-transformed dashboard shallow feature map along the channel dimension to obtain a first branch feature map, a second branch feature map, a third branch feature map, and a fourth branch feature map; a feature extraction unit for passing the first branch feature map through a convolutional neural network model in the dashboard multi-scale feature enhancement and expression module to obtain a first branch output feature map; a second convolutional unit for processing the second branch feature map through a second convolutional layer with a 3×3 convolutional kernel in the dashboard multi-scale feature enhancement and expression module to obtain a second branch output feature map; a third convolutional unit for fusing the second branch output feature map and the third branch feature map and then processing them through a third convolutional layer with a 3×3 convolutional kernel in the dashboard multi-scale feature enhancement and expression module to obtain a third branch output feature map; a fourth convolutional unit for fusing the third branch output feature map and the fourth branch feature map and then processing them through a fourth convolutional layer with a 3×3 convolutional kernel in the dashboard multi-scale feature enhancement and expression module to obtain a fourth branch output feature map; a multi-branch fusion unit for fusing the first branch output feature map, the second branch output feature map, the third branch output feature map, and the fourth branch output feature map to obtain a multi-branch fusion feature map; a fifth convolutional unit for processing the multi-branch fusion feature map through a fifth convolutional layer with a 1×1 convolutional kernel in the dashboard multi-scale feature enhancement and expression module to obtain a channel-transformed multi-branch fusion feature map; a multi-scale fusion unit for fusing the channel-transformed multi-branch fusion feature map and the dashboard shallow feature map to obtain the dashboard multi-scale shallow feature map.

[0014] Optionally, the dashboard semantic information transfer and highlighting shallow feature module includes: a dashboard multi-scale semantic space focusing unit for calculating a spatial attention feature matrix of the dashboard multi-scale semantic feature map; a masking processing unit for performing masking processing on the spatial attention feature matrix based on a predetermined threshold to obtain a masked spatial attention feature matrix; and a dashboard shallow feature foreground highlighting unit for calculating the element-wise multiplication between each feature matrix of the dashboard multi-scale shallow feature map and the masked spatial attention feature matrix to obtain the foreground highlighted dashboard shallow feature map as the foreground highlighted dashboard shallow feature.

[0015] Optionally, the dashboard defect detection module is configured to: pass the foreground highlighted dashboard shallow feature map through a dashboard defect recognizer based on a classifier to obtain an identification result, where the identification result is used to indicate whether there is a defect in the dashboard to be detected.

[0016] Optionally, it further includes a training module for training the dashboard feature extractor based on the pyramid network, the dashboard multi-scale feature enhancement and expression module, and the dashboard defect recognizer based on the classifier.

[0017] Optionally, the training module includes: a training dashboard image acquisition unit for acquiring a training detection image of the detected dashboard collected by a camera; a training dashboard multi-level feature extraction unit for performing multi-level feature extraction on the training detection image of the detected dashboard through the dashboard feature extractor based on a deep neural network to obtain a training dashboard shallow feature map and a training dashboard semantic feature map; a training dashboard feature enhanced expression unit for passing the training dashboard shallow feature map and the training dashboard semantic feature map through the dashboard multi-scale feature enhanced expression module to obtain a training dashboard multi-scale shallow feature map and a training dashboard multi-scale semantic feature map; a training dashboard multi-scale semantic space focusing unit for calculating a training spatial attention feature matrix of the training dashboard multi-scale semantic feature map; a training masking processing unit for performing masking processing on the training spatial attention feature matrix based on a predetermined threshold to obtain a training masked spatial attention feature matrix; a training dashboard shallow feature foreground highlighting unit for calculating the product of each feature matrix of the training dashboard multi-scale shallow feature map and the training masked spatial attention feature matrix at each position point to obtain a training foreground highlighted dashboard shallow feature map; a training classification unit for passing the training foreground highlighted dashboard shallow feature map through the dashboard defect identifier based on a classifier to obtain a training classification loss function value; a training unit for training the dashboard feature extractor based on a pyramid network, the dashboard multi-scale feature enhanced expression module, and the dashboard defect identifier based on a classifier using the weighted sum of the training classification loss function values as a loss function value, and optimizing the training foreground highlighted dashboard shallow feature map at each model iteration.

[0018] In a second aspect, the present invention provides a method for final inspection of a dashboard, the method including:

[0019] Acquiring a detection image of the detected dashboard collected by a camera;

[0020] Performing multi-level feature extraction on the detection image of the detected dashboard through a dashboard feature extractor based on a deep neural network to obtain a dashboard shallow feature map and a dashboard semantic feature map;

[0021] Passing the dashboard shallow feature map and the dashboard semantic feature map through a dashboard multi-scale feature enhanced expression module to obtain a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map;

[0022] Based on the dashboard multi-scale semantic feature map, performing region transfer based on an attention mechanism on the dashboard multi-scale shallow feature map to obtain a foreground highlighted dashboard shallow feature;

[0023] Based on the foreground highlighted dashboard shallow features, determine whether there are defects in the detected dashboard.

[0024] Optionally, the dashboard feature extractor based on the deep neural network is a dashboard feature extractor based on the pyramid network.

[0025] With the above technical solution, through the dashboard feature extractor based on the deep neural network, multi-level feature extraction is performed on the detected image of the dashboard to obtain the dashboard shallow feature map and the dashboard semantic feature map; the dashboard shallow feature map and the dashboard semantic feature map are passed through the dashboard multi-scale feature enhancement expression module to obtain the dashboard multi-scale shallow feature map and the dashboard multi-scale semantic feature map; based on the dashboard multi-scale semantic feature map, region transfer based on the attention mechanism is performed on the dashboard multi-scale shallow feature map to obtain the foreground highlighted dashboard shallow features, so as to determine whether there are defects in the detected dashboard. In this way, intelligent final inspection of the dashboard can be realized, thereby avoiding the errors and low efficiency problems brought by the traditional manual final inspection method, being able to well handle complex dashboard defect problems, and ensuring that the quality of the dashboard product meets the standard requirements.

[0026] Other features and advantages of the present invention will be described in detail in the subsequent specific implementation part. Brief Description of the Drawings

[0027] Combined with the drawings and referring to the following specific implementation manners, the above and other features, advantages and aspects of the embodiments of the present invention will become more obvious. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic, and the original parts and elements are not necessarily drawn to scale. In the drawings:

[0028] Figure 1 is a block diagram of a dashboard final inspection system shown according to an exemplary embodiment.

[0029] Figure 2 is a flowchart of a dashboard final inspection method shown according to an exemplary embodiment.

[0030] Figure 3 is a block diagram of an electronic device shown according to an exemplary embodiment.

[0031] Figure 4 is an application scenario diagram of a dashboard final inspection system shown according to an exemplary embodiment. Detailed Description of the Invention

[0032] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not used to limit the scope of protection of the present invention.

[0033] It should be understood that the various steps recited in the method embodiments of the present invention can be executed in a different order and / or executed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0034] The term "comprising" and its variations used herein are open-ended, i.e., "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0035] It should be noted that the concepts such as "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependent relationships.

[0036] It should be noted that the modifications of "one" and "plural" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise clearly specified in the context, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0038] To solve the above problems, the present invention provides a dashboard final inspection system and method. The detection image of the dashboard to be detected is subjected to multi-level feature extraction by a dashboard feature extractor based on a deep neural network to obtain a dashboard shallow feature map and a dashboard semantic feature map. The dashboard shallow feature map and the dashboard semantic feature map are passed through a dashboard multi-scale feature enhancement and expression module to obtain a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map. Based on the dashboard multi-scale semantic feature map, region transfer based on an attention mechanism is performed on the dashboard multi-scale shallow feature map to obtain a foreground highlighted dashboard shallow feature, so as to determine whether there are defects in the dashboard to be detected. In this way, intelligent final inspection of the dashboard can be realized, thereby avoiding the errors and low efficiency problems brought by the traditional manual final inspection method, being able to well handle complex dashboard defect problems, and ensuring that the dashboard product quality meets the standard requirements.

[0039] The following will describe in detail the specific embodiments of the present invention with reference to the accompanying drawings.

[0040] Figure 1 is a block diagram of a dashboard final inspection system shown according to an exemplary embodiment. As Figure 1 shown, the system 100 includes:

[0041] A dashboard image acquisition module 101, configured to acquire a detection image of the dashboard to be detected collected by a camera;

[0042] A dashboard multi-level feature extraction module 102, configured to perform multi-level feature extraction on the detection image of the dashboard to be detected by a dashboard feature extractor based on a deep neural network to obtain a dashboard shallow feature map and a dashboard semantic feature map;

[0043] A dashboard feature enhancement and expression module 103, configured to pass the dashboard shallow feature map and the dashboard semantic feature map through a dashboard multi-scale feature enhancement and expression module to obtain a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map;

[0044] A dashboard semantic information transfer and highlighted shallow feature module 104, configured to perform region transfer based on an attention mechanism on the dashboard multi-scale shallow feature map based on the dashboard multi-scale semantic feature map to obtain a foreground highlighted dashboard shallow feature;

[0045] A dashboard defect detection module 105, configured to determine whether there are defects in the dashboard to be detected based on the foreground highlighted dashboard shallow feature.

[0046] Wherein, the dashboard feature extractor based on a deep neural network is a dashboard feature extractor based on a pyramid network.

[0047] It should be understood that machine vision is a technology that uses computer algorithms and image processing techniques to analyze and understand images and videos. In the dashboard final inspection system, machine vision technology can help identify key components, detect the position of the pointer, and detect abnormal conditions, such as pointer deviation or abnormal display. Therefore, in the process of final inspection of the dashboard, the application of machine vision technology is crucial.

[0048] Based on this, the technical concept of the present invention is to collect the final inspection image of the dashboard through a camera, and introduce image processing and analysis algorithms based on artificial intelligence and machine vision technology at the backend to analyze this image, so as to capture the multi-scale and multi-faceted fusion features at different levels of the dashboard, thereby automatically identifying whether there are abnormalities in key components such as the pointer position and scale of the dashboard, such as whether there are problems such as pointer deviation or abnormal display. In this way, the intelligent final inspection of the dashboard can be realized, thus avoiding the errors and low efficiency problems brought by the traditional manual final inspection method, being able to well handle complex dashboard defect problems, and ensuring that the quality of the dashboard product meets the standard requirements.

[0049] Specifically, in the technical solution of the present invention, first, obtain the detection image of the dashboard to be detected collected by the camera. Then, considering that the detection image of the dashboard to be detected contains a large amount of information, some of which is very important for detecting the quality and abnormalities of the dashboard, while some information is noise interference or irrelevant. And, it is also considered that in the actual process of quality inspection of the dashboard, not only the shallow features such as the edge, texture, and color of the dashboard need to be concerned, which helps to judge problems such as pointer shape defects, scale color defects, and dashboard edge missing in the dashboard, but also the deep semantic features of the dashboard need to be concerned, such as the semantic information represented by the scale. Based on this, in the technical solution of the present invention, further pass the detection image of the dashboard to be detected through a dashboard feature extractor based on a pyramid network to obtain a dashboard shallow feature map and a dashboard semantic feature map. Through feature mining by the dashboard feature extractor based on the pyramid network, it is possible to capture feature information at different levels of the dashboard surface. Among them, the shallow features usually contain more detailed information and surface visual features, while the semantic features are more abstract and have semantic feature information. This hierarchical feature representation helps to capture different levels of features of the dashboard surface in the image, improve the model's understanding ability of the image content of the dashboard, and thus realize the accurate identification and detection of the dashboard.

[0050] Then, considering that different defect problems of the dashboard are often reflected at different levels and scales of the image, some features and defects may be more easily recognizable at a smaller scale, such as the color of a specific area of the dashboard and the degree of bending of the dashboard pointer, while some features and defects need to be accurately captured at a larger scale, such as the overall shape of the dashboard. Based on this, in order to make full use of the different-level and multi-scale feature information in the image to enhance the expression of the dashboard surface features, thereby improving the model's understanding ability and defect detection ability for the dashboard image, in the technical solution of the present invention, the dashboard shallow feature map and the dashboard semantic feature map are further passed through the dashboard multi-scale feature enhancement expression module to obtain a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map. The dashboard multi-scale feature enhancement expression module can respectively perform multi-scale feature capture on the shallow features and deep semantic features related to the dashboard in the dashboard shallow feature map and the dashboard semantic feature map, so that the feature information at different scales in the shallow and deep features of the dashboard can be comprehensively utilized, improving the richness and diversity of the dashboard feature expression, enabling the model to better understand the content in the image, and thus improving the accuracy of dashboard defect detection and problem recognition.

[0051] In one embodiment of the present invention, the dashboard feature enhanced expression module includes: a first convolutional unit, configured to obtain a channel-transformed dashboard shallow feature map by passing the dashboard shallow feature map through a first convolutional layer based on a 1×1 convolutional kernel in the dashboard multi-scale feature enhanced expression module; a feature map splitting unit, configured to split the channel-transformed dashboard shallow feature map along the channel dimension to obtain a first branch feature map, a second branch feature map, a third branch feature map, and a fourth branch feature map; a feature extraction unit, configured to obtain a first branch output feature map by passing the first branch feature map through a convolutional neural network model in the dashboard multi-scale feature enhanced expression module; a second convolutional unit, configured to process the second branch feature map through a second convolutional layer based on a 3×3 convolutional kernel in the dashboard multi-scale feature enhanced expression module to obtain a second branch output feature map; a third convolutional unit, configured to process the second branch output feature map and the third branch feature map after fusion through a third convolutional layer based on a 3×3 convolutional kernel in the dashboard multi-scale feature enhanced expression module to obtain a third branch output feature map; a fourth convolutional unit, configured to process the third branch output feature map and the fourth branch feature map after fusion through a fourth convolutional layer based on a 3×3 convolutional kernel in the dashboard multi-scale feature enhanced expression module to obtain a fourth branch output feature map; a multi-branch fusion unit, configured to fuse the first branch output feature map, the second branch output feature map, the third branch output feature map, and the fourth branch output feature map to obtain a multi-branch fusion feature map; a fifth convolutional unit, configured to process the multi-branch fusion feature map through a fifth convolutional layer based on a 1×1 convolutional kernel in the dashboard multi-scale feature enhanced expression module to obtain a channel-transformed multi-branch fusion feature map; a multi-scale fusion unit, configured to fuse the channel-transformed multi-branch fusion feature map and the dashboard shallow feature map to obtain the dashboard multi-scale shallow feature map.

[0052] Furthermore, since the multi-scale semantic feature map of the dashboard contains the multi-scale deep semantic features of the dashboard surface, which reflect the deep surface semantics of different aspects of the dashboard, such as the scale semantics, pointer offset semantics, and display semantics of the dashboard, these deep semantic features are particularly important for defect detection. The multi-scale shallow feature map of the dashboard contains the multi-scale shallow features of the dashboard surface, which reflect the shallow features of different aspects of the dashboard surface, such as the colors of each area on the dashboard surface, the shape, and texture information of the dashboard. These feature information have an interrelated effect and are jointly crucial for improving the accuracy of dashboard defect detection. Based on this, in the technical solution of the present invention, further based on the multi-scale semantic feature map of the dashboard, region transfer based on the attention mechanism is performed on the multi-scale shallow feature map of the dashboard to obtain a foreground-prominent dashboard shallow feature map. It should be understood that by performing region transfer processing based on the attention mechanism on the multi-scale shallow feature map of the dashboard, the high-level semantic features of the dashboard can be transferred to the multi-scale shallow feature map of the dashboard to highlight the foreground semantic part of the multi-scale shallow feature map of the dashboard, which is related to the quality inspection of the dashboard. Moreover, in the process of transferring deep semantic features to the shallow feature map, by performing attention weighting on the deep semantic multi-scale representation features to highlight the foreground part of the shallow features in the image, the attention to the key information related to dashboard defect detection is improved, thereby improving the capture and expression ability of the key information and optimizing the defect detection performance of the dashboard.

[0053] In an embodiment of the present invention, the dashboard semantic information transfer and prominent shallow feature module includes: a dashboard multi-scale semantic space focusing unit for calculating the spatial attention feature matrix of the multi-scale semantic feature map of the dashboard; a masking processing unit for performing masking processing on the spatial attention feature matrix based on a predetermined threshold to obtain a masked spatial attention feature matrix; and a dashboard shallow feature foreground prominent unit for calculating the product of each feature matrix of the multi-scale shallow feature map of the dashboard and the masked spatial attention feature matrix at the corresponding position points to obtain the foreground-prominent dashboard shallow feature map as the foreground-prominent dashboard shallow feature.

[0054] Subsequently, the foreground-emphasized dashboard shallow feature map is then passed through a dashboard defect recognizer based on a classifier to obtain a recognition result, which is used to indicate whether there are defects in the detected dashboard. That is to say, the classified processing is carried out using the foreground-emphasized shallow semantic features of the dashboard, so as to identify the defects of the dashboard, thereby automatically identifying whether there are abnormalities in key components such as the pointer position of the dashboard, for example, whether there are problems such as pointer deviation or display abnormality. In this way, the intelligent final inspection of the dashboard can be realized, thus avoiding the errors and low efficiency problems brought by manual inspection in the traditional final inspection scheme, and being able to well handle complex dashboard defect problems, ensuring that the quality of the dashboard product meets the standard requirements.

[0055] In an embodiment of the present invention, the dashboard defect detection module is configured to: pass the foreground-emphasized dashboard shallow feature map through a dashboard defect recognizer based on a classifier to obtain a recognition result, and the recognition result is used to indicate whether there are defects in the detected dashboard.

[0056] Further, in an embodiment of the present invention, the dashboard final inspection system further includes a training module for training the dashboard feature extractor based on the pyramid network, the dashboard multi-scale feature enhancement expression module, and the dashboard defect identifier based on the classifier. The training module includes: a training dashboard image acquisition unit for acquiring training detection images of the dashboard to be detected collected by a camera; a training dashboard multi-level feature extraction unit for performing multi-level feature extraction on the training detection images of the dashboard to be detected through the dashboard feature extractor based on the deep neural network to obtain a training dashboard shallow feature map and a training dashboard semantic feature map; a training dashboard feature enhancement expression unit for passing the training dashboard shallow feature map and the training dashboard semantic feature map through the dashboard multi-scale feature enhancement expression module to obtain a training dashboard multi-scale shallow feature map and a training dashboard multi-scale semantic feature map; a training dashboard multi-scale semantic space focusing unit for calculating a training spatial attention feature matrix of the training dashboard multi-scale semantic feature map; a training masking processing unit for performing masking processing on the training spatial attention feature matrix based on a predetermined threshold to obtain a training masked spatial attention feature matrix; a training dashboard shallow feature foreground highlighting unit for calculating the product of each feature matrix of the training dashboard multi-scale shallow feature map and the training masked spatial attention feature matrix at the position points to obtain a training foreground highlighted dashboard shallow feature map; a training classification unit for passing the training foreground highlighted dashboard shallow feature map through the dashboard defect identifier based on the classifier to obtain a training classification loss function value; a training unit for training the dashboard feature extractor based on the pyramid network, the dashboard multi-scale feature enhancement expression module, and the dashboard defect identifier based on the classifier with the weighted sum of the training classification loss function values as the loss function value, and optimizing the training foreground highlighted dashboard shallow feature map at each model iteration.

[0057] In the technical solution described above, the training dashboard shallow feature map and the training dashboard semantic feature map respectively represent the image semantic features of different scales and different depths based on the pyramid network of the training detection images of the dashboard to be detected. Thus, after passing the training dashboard shallow feature map and the training dashboard semantic feature map through the dashboard multi-scale feature enhancement expression module, the obtained training dashboard multi-scale shallow feature map and the training dashboard multi-scale semantic feature map also have differences in image semantic feature patterns based on different feature scale representations and feature depth representations.

[0058] Therefore, when considering the region transfer based on the attention mechanism for the multi-scale shallow feature maps of the training instrument panel using the multi-scale semantic feature maps of the training instrument panel, under the cascade model branch structure for image semantic feature extraction with different scales and depths, if the corresponding relationship of image semantic features between the multi-scale semantic feature maps of the training instrument panel and then the multi-scale shallow feature maps of the training instrument panel can be suppressed, the classification iteration effect can be improved.

[0059] Based on this, in each model iteration of the present application, the shallow feature map of the training foreground prominent instrument panel is optimized, including the steps of: expanding the shallow feature map of the training foreground prominent instrument panel into a shallow feature vector of the training foreground prominent instrument panel; determining the class probability value obtained by passing the shallow feature vector of the training foreground prominent instrument panel through a classifier and the probability difference of one minus the class probability value; calculating the relative class probability value obtained by dividing the class probability value by the probability difference; calculating the collaborative class probability value obtained by multiplying the class probability value by the probability difference; determining the first eigenvalue and the second eigenvalue at any two different positions of the shallow feature vector of the training foreground prominent instrument panel; multiplying the first eigenvalue and the second eigenvalue after subtracting the relative class probability value respectively to obtain a first intermediate eigenvalue; adding the first eigenvalue and the second eigenvalue and then dividing by the collaborative class probability value to obtain a second intermediate eigenvalue; adding the first intermediate eigenvalue and the second intermediate eigenvalue, and using the result as the matrix value at the coordinate corresponding to the different positions of the first eigenvalue and the second eigenvalue to obtain a correction matrix; calculating the matrix product of the correction matrix and the shallow feature vector of the training foreground prominent instrument panel to obtain an optimized shallow feature vector of the training foreground prominent instrument panel, where the shallow feature vector of the training foreground prominent instrument panel is a column vector; and restoring the optimized shallow feature vector of the training foreground prominent instrument panel to an optimized shallow feature map of the training foreground prominent instrument panel.

[0060] That is, by taking the eigenvalue of the training foreground highlighted dashboard shallow feature map as a unit, for the relative class probability distribution form and the collaborative class probability distribution form of the class probability value and the probability difference obtained by the classifier for the training foreground highlighted dashboard shallow feature map as a feature set, the relative-based joint representation and the collaboration-based response representation of the eigenvalue pairs of the training foreground highlighted dashboard shallow feature map are respectively performed, so as to avoid the class induction bias caused by the corresponding local distribution differences between the eigenvalues of the training foreground highlighted dashboard shallow feature map, and to establish a robust understanding paradigm for class recognition of the training foreground highlighted dashboard shallow feature map as a whole feature set, thereby improving the iterative effect of the training foreground highlighted dashboard shallow feature map as a feature set in the classification process. That is, it improves the classification training speed and the accuracy of the classification result when the training foreground highlighted dashboard shallow feature map is classified by the classifier. In this way, the intelligent final inspection of the dashboard can be more accurately realized, thereby avoiding the errors and low efficiency problems brought by the traditional final inspection scheme relying on manual detection, being able to well handle complex dashboard defect problems, and ensuring that the dashboard product quality meets the standard requirements.

[0061] In summary, adopting the above solution, the final inspection image of the dashboard is collected by the camera, and an image processing and analysis algorithm based on artificial intelligence and machine vision technology is introduced at the backend to analyze the image, so as to capture the multi-scale and multi-faceted fusion features at different levels of the dashboard, thereby automatically identifying whether there are abnormalities in key components such as the pointer position and scale of the dashboard, such as whether there are problems such as pointer deviation or display abnormality. In this way, the intelligent final inspection of the dashboard can be realized, thereby avoiding the errors and low efficiency problems brought by the traditional manual final inspection method, being able to well handle complex dashboard defect problems, and ensuring that the dashboard product quality meets the standard requirements.

[0062] Figure 2 It is a flowchart of a dashboard final inspection method shown according to an exemplary embodiment, as Figure 2 shown, the method includes:

[0063] Step 201, obtain a detection image of the dashboard to be detected collected by the camera;

[0064] Step 202, perform multi-level feature extraction on the detection image of the dashboard to be detected through a dashboard feature extractor based on a deep neural network to obtain a dashboard shallow feature map and a dashboard semantic feature map;

[0065] Step 203, pass the dashboard shallow feature map and the dashboard semantic feature map through a dashboard multi-scale feature enhancement expression module to obtain a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map;

[0066] Step 204: Based on the dashboard multi-scale semantic feature map, perform region transfer based on the attention mechanism on the dashboard multi-scale shallow feature map to obtain a foreground highlighted dashboard shallow feature;

[0067] Step 205: Based on the foreground highlighted dashboard shallow feature, determine whether there are defects in the dashboard to be detected.

[0068] In an embodiment of the present invention, the dashboard feature extractor based on the deep neural network is a dashboard feature extractor based on the pyramid network.

[0069] Reference is made below to Figure 3 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing the embodiments of the present invention. The terminal device in the embodiments of the present invention may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present invention.

[0070] As Figure 3 shown, the electronic device 600 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0071] Generally, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 3 shows the electronic device 600 having various devices, it should be understood that it is not required to implement or include all the shown devices. More or fewer devices may be alternatively implemented or included.

[0072] In particular, according to an embodiment of the present invention, the processes described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above functions defined in the method of the embodiment of the present invention are performed.

[0073] It should be noted that the above computer-readable medium in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device. In the present invention, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and the computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0074] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LANs"), wide area networks ("WANs"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0075] The above computer-readable medium can be included in the above electronic device; or it can exist separately without being assembled into the electronic device.

[0076] Computer program code for performing the operations of the present invention can be written in one or more programming languages or combinations thereof. The above programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, and C++; and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0078] The modules involved in the embodiments of the present invention can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the module itself in some cases. For example, the test parameter acquisition module can also be described as "the module for acquiring device test parameters corresponding to the target device".

[0079] The functions described above in this article can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Product (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0080] In the context of the present invention, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM or Flash Memory), optical fiber, portable compact disc read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0081] Figure 4 is an application scenario diagram of a dashboard final inspection system shown according to an exemplary embodiment. As Figure 4 shown, in this application scenario, first, a detection image of the dashboard to be detected collected by a camera is obtained (for example, as Figure 4 shown by C); then, the obtained detection image is input into a server (for example, as Figure 4 shown by S) deployed with a dashboard final inspection algorithm, where the server can process the detection image based on the dashboard final inspection algorithm to determine whether there are defects in the dashboard to be detected.

[0082] The above description is only a preferred embodiment of the present invention and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of disclosure involved in the present invention is not limited to the technical solutions formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosure concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present invention.

[0083] In addition, although the operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments.

[0084] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. On the contrary, the specific features and acts described above are merely example forms for implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

Claims

1. A final inspection system for an instrument panel, characterized in that: include: The instrument panel image acquisition module is used to obtain the detection image of the instrument panel to be detected collected by the camera; A multi-level feature extraction module for a dashboard, used for performing multi-level feature extraction on the detection image of the detected dashboard through a dashboard feature extractor based on a deep neural network to obtain a shallow feature map of the dashboard and a semantic feature map of the dashboard; A dashboard feature enhancement expression module, used for obtaining a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map by passing the dashboard shallow feature map and the dashboard semantic feature map through a dashboard multi-scale feature enhancement expression module; A dashboard semantic information transfer and shallow feature highlighting module is used to perform an attention mechanism-based regional transfer on the dashboard multi-scale shallow feature map based on the dashboard multi-scale semantic feature map to obtain a foreground-highlighted dashboard shallow feature; An instrument panel defect detection module, used to determine whether the instrument panel under inspection has defects based on the shallow features of the instrument panel highlighted by the foreground; The dashboard semantic information transfer highlights the shallow feature modules, including: A multi-scale semantic spatial focusing unit for a dashboard, used to calculate a spatial attention feature matrix of the multi-scale semantic feature map of the dashboard; a masking processing unit, configured to perform masking processing on the spatial attention feature matrix based on a predetermined threshold to obtain a masked spatial attention feature matrix; A dashboard shallow feature foreground highlighting unit, used for calculating the position point multiplication between each feature matrix of the dashboard multi-scale shallow feature map and the masked spatial attention feature matrix to obtain the foreground highlighted dashboard shallow feature map as the foreground highlighted dashboard shallow feature; Also included is a training module for training a pyramid network-based instrument panel feature extractor, the instrument panel multi-scale feature enhancement expression module, and a classifier-based instrument panel defect identifier; The training module comprises: A training instrument panel image acquisition unit, used to acquire a training detection image of the instrument panel to be detected acquired by a camera; A training dashboard multi-level feature extraction unit, used for performing multi-level feature extraction on the training detection image of the detected dashboard through the dashboard feature extractor based on the deep neural network to obtain a training dashboard shallow feature map and a training dashboard semantic feature map; A training dashboard feature enhancement expression unit, used for passing the training dashboard shallow feature map and the training dashboard semantic feature map through the dashboard multi-scale feature enhancement expression module to obtain a training dashboard multi-scale shallow feature map and a training dashboard multi-scale semantic feature map; A training dashboard multi-scale semantic space focusing unit, used to calculate a training space attention feature matrix of the training dashboard multi-scale semantic feature map; A training masking processing unit, configured to perform masking processing on the training spatial attention feature matrix based on a predetermined threshold value to obtain a training masked spatial attention feature matrix; A training instrument panel shallow feature foreground highlighting unit is used to calculate the position point multiplication between each feature matrix of the training instrument panel multi-scale shallow feature map and the training masked spatial attention feature matrix to obtain a training foreground highlighting instrument panel shallow feature map; A training classification unit, used for passing the training foreground highlighting instrument panel shallow feature map through the classifier-based instrument panel defect identifier to obtain a training classification loss function value; A training unit is used to train the pyramid network-based dashboard feature extractor, the dashboard multi-scale feature enhancement expression module and the classifier-based dashboard defect identifier based on the weighted sum of the training classification loss function values ​​as the loss function value, and optimize the training foreground-highlighting dashboard shallow feature map at each model iteration.

2. The instrument panel final inspection system according to claim 1, characterized in that: The dashboard feature enhancement expression module includes: A first convolution unit, used for passing the shallow feature map of the instrument panel through the first convolution layer based on a 1×1 convolution kernel in the multi-scale feature enhancement expression module of the instrument panel to obtain a channel-transformed shallow feature map of the instrument panel; A feature map splitting unit, used for splitting the shallow feature map of the channel transformation instrument panel along the channel dimension to obtain a first branch feature map, a second branch feature map, a third branch feature map and a fourth branch feature map; A feature extraction unit, configured to pass the first branch feature map through a convolutional neural network model in the dashboard multi-scale feature enhancement expression module to obtain a first branch output feature map; A second convolution unit, used for processing the second branch feature map through a second convolution layer based on a 3×3 convolution kernel in the dashboard multi-scale feature enhancement expression module to obtain a second branch output feature map; A third convolution unit, used for fusing the second branch output feature map and the third branch feature map, and processing the resultant result through a third convolution layer based on a 3×3 convolution kernel in the dashboard multi-scale feature enhancement expression module to obtain a third branch output feature map; a fourth convolution unit, configured to fuse the third branch output feature map and the fourth branch feature map and process the merged feature map through a fourth convolution layer based on a 3×3 convolution kernel in the multi-scale feature enhancement expression module of the instrument panel to obtain a fourth branch output feature map; A multi-branch fusion unit, used to fuse the first branch output feature map, the second branch output feature map, the third branch output feature map and the fourth branch output feature map to obtain a multi-branch fusion feature map; A fifth convolution unit, used for processing the multi-branch fusion feature map through a fifth convolution layer based on a 1×1 convolution kernel in the dashboard multi-scale feature enhancement expression module to obtain a channel transformation multi-branch fusion feature map; The multi-scale fusion unit is used to fuse the channel transformation multi-branch fusion feature map and the instrument panel shallow feature map to obtain the instrument panel multi-scale shallow feature map.

3. A final inspection method for an instrument panel using the final inspection system for an instrument panel according to any one of claims 1 to 2, characterized in that: include: Acquire a detection image of the detected instrument panel captured by a camera; Performing multi-level feature extraction on the detection image of the detected dashboard by a dashboard feature extractor based on a deep neural network to obtain a dashboard shallow feature map and a dashboard semantic feature map; The dashboard shallow feature map and the dashboard semantic feature map are passed through a dashboard multi-scale feature enhancement expression module to obtain a dashboard multi-scale shallow feature map and a dashboard multi-scale semantic feature map; Based on the multi-scale semantic feature map of the dashboard, performing a region transfer based on the attention mechanism on the multi-scale shallow feature map of the dashboard to obtain shallow features of the dashboard with a foreground highlight; Based on the foreground highlighting the shallow features of the instrument panel, it is determined whether the inspected instrument panel has defects.

Images