Test state auxiliary monitoring method and equipment based on machine vision

Abstract

The invention relates to the technical field of image processing, in particular to a test state auxiliary monitoring method and device based on machine vision. The method comprises the following steps: performing real-time shooting in a test process of an automobile sample piece to obtain multiple frames of images and a corresponding relation between each frame of image and test data; calculating the difference between the target frame image and the reference frame in the multiple frames of images to obtain a first difference chart; performing contour recognition on the first difference image to obtain an area of a dynamic target object and a first pixel number of the dynamic target object; taking an intersection of the area of the dynamic target object and the reference area to obtain the proportion of the intersection area in the dynamic target object; calculating the maximum number of pixels of the dynamic target object according to the reference frame and a full-amount image corresponding to the state change of the test full-amount data; and if the proportion is greater than a set threshold value and the first pixel number is less than the maximum pixel number, reserving the target frame image. According to the invention, the test state can be automatically and accurately monitored.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for auxiliary monitoring of test status based on machine vision. Background Art

[0002] Automotive electronic reliability test equipment is used to test the performance and durability of automotive electronic components, systems or vehicles under specific environmental conditions. These devices can simulate various environmental conditions that the tested samples may encounter in actual use, such as temperature, humidity, vibration, shock, electromagnetic interference, etc., so as to evaluate their reliability and stability.

[0003] Common automotive electronic reliability test equipment includes: high and low temperature test chambers, damp heat test chambers, vibration test benches, impact test machines, and electromagnetic compatibility (EMC) test equipment. These devices are generally equipped with status indicators and screens. The status indicator uses light to monitor the working or position status of circuits and electrical equipment, and the screen displays the test data and environmental parameters generated during the test in real time.

[0004] With the rapid development of automotive intelligence, the testing requirements in the field of automotive electronic reliability and environmental reliability are also increasing. Most tests require personnel to monitor the test status changes and the movement of the tested samples 24 hours a day and record them in real time. The labor cost is high and it is easy to cause negligence and omissions. Summary of the invention

[0005] The purpose of this application is to provide a test status auxiliary monitoring method and device based on machine vision to automatically and accurately monitor the test status.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a test state auxiliary monitoring method based on machine vision, comprising:

[0008] Real-time shooting is performed during the test of automobile samples to obtain multiple frames of images and the corresponding relationship between each frame of image and test data;

[0009] Calculating the difference between the target frame image and the reference frame in the multiple frame images to obtain a first difference map; the reference frame corresponds to a test preparation state; the target frame image is any frame image in the multiple frame images;

[0010] Performing contour recognition on the first difference map to obtain an area of ​​the dynamic target and a first number of pixels of the dynamic target;

[0011] Perform dynamic region recognition on multiple frames of non-interference and non-jitter test images, and take the union of the identified dynamic regions to obtain the reference region;

[0012] Taking the intersection of the area of ​​the dynamic target object and the reference area to obtain the proportion of the intersection area in the dynamic target object;

[0013] Calculate the maximum number of pixels of the dynamic target according to the full image corresponding to the state change of the reference frame and the full test data;

[0014] If the ratio is greater than a set threshold and the first number of pixels is less than the maximum number of pixels, retain the target frame image;

[0015] The dynamic target is captured according to the pixel difference between adjacent frames in the retained image, and the test state displayed by the dynamic target is obtained.

[0016] In a second aspect, the present application provides an electronic device, including:

[0017] at least one processor, and a memory communicatively coupled to at least one of the processors;

[0018] Wherein, the memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute any of the machine vision-based test status auxiliary monitoring methods.

[0019] Compared with the prior art, the beneficial effects of this application are:

[0020] This application improves the intelligence level of the laboratory and realizes automatic monitoring by replacing manpower with machines. The test status auxiliary monitoring method based on machine vision can record information in real time, capture photos of the status changes of the monitored samples, free up manpower, and objectively record the test status. At the same time, it solves false alarms caused by external interference, accidental capture of dynamic images and other perimeter intrusion problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flow chart of a test state auxiliary monitoring method based on machine vision provided in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of a reference area provided in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a selected area provided in an embodiment of the present application;

[0025] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] Figure 1 This is a flow chart of a test status auxiliary monitoring method based on machine vision provided by this embodiment. This embodiment is applicable to the scene where the state of the test equipment and the action of the sample are photographed to obtain the test status during any test of the automobile sample. This method can be executed by an electronic device. The electronic device can be integrated in the vehicle computer or can be independent of the vehicle computer. Figure 1 As shown, this embodiment provides a test state auxiliary monitoring method based on machine vision, comprising the following steps:

[0028] S110, performing real-time shooting during the test of the automobile sample to obtain multiple frames of images and the corresponding relationship between each frame of image and the test data.

[0029] At least one camera is set up at the test site of the automobile sample, facing the test equipment and the automobile sample, so as to capture images in real time according to a preset shooting cycle during the test of the automobile sample. During the test, the test equipment itself will also record test data, such as ambient temperature, deformation of the automobile sample, etc.

[0030] The test equipment includes status indicator lights and screens. The status indicator lights use lights to monitor the working or position status of circuits and electrical equipment, and the screen displays test data and environmental parameters generated during the test in real time.

[0031] S120, calculating the difference between the target frame image and the reference frame in the multiple frame images to obtain a first difference map.

[0032] For the convenience of description and distinction, any frame image in the multiple frame image reference frame is called the target frame image, that is, the following operations S120 to S180 will be performed for each frame image. The reference frame corresponds to the test preparation state, that is, the test data is the data during the test preparation, and the sample is in the state during the test preparation, that is, there is no abnormality. The status indicator lights are all green, there is no alarm on the screen, and there is no test data.

[0033] Optionally, both the target frame image and the reference frame are converted into grayscale images to obtain a target frame grayscale image and a reference frame grayscale image, respectively. Converting each frame of the image into a grayscale image and applying Gaussian blur to reduce noise can improve the stability of subsequent processing. After grayscale processing, different pixel values ​​will be reflected in the difference in grayscale, then the grayscale values ​​of the pixels at the same position in the target frame grayscale image and the reference frame grayscale image are subtracted respectively to obtain a first difference map. The first difference map reflects the changes in the target frame image relative to the test preparation state. This change may be caused by the test itself (such as display changes of the test equipment or deformation / displacement of the sample), or it may be caused by external interference (such as interference objects such as people or flying objects intervening in the image) or image jitter.

[0034] S130 , performing contour recognition on the first difference map to obtain a region of the dynamic target object and a first number of pixels of the dynamic target object.

[0035] First, the first difference map is converted into a binary map.

[0036] The first difference map here can be a grayscale image. A binary image is a special grayscale image with only two grayscale levels: black and white, with no filter value in between. In a binary image, each pixel has only two possible values, usually 0 for black and 1 for white (or vice versa). Specifically, a brightness threshold is set; each pixel of the grayscale image is traversed and its grayscale value is compared with the threshold. If the grayscale value is greater than or equal to the threshold, it is set to the maximum grayscale value (usually 255, i.e. white); if the grayscale value is less than the threshold, it is set to the minimum grayscale value (usually 0, i.e. black).

[0037] Next, contour recognition is performed on the binary image to obtain the area of ​​the dynamic target object.

[0038] Optionally, the contour detection function of OpenCV is used to extract the contour of the region of the dynamic target object so as to further analyze its size and position. The dynamic target object here is relative to the test preparation state.

[0039] Finally, the binary image with the identified contour is expanded to expand the area of ​​the dynamic target, and the area of ​​the dynamic target and the first pixel number of the dynamic target in the expanded image are obtained.

[0040] The dilation operation is used to increase the size of the dynamic target in the binary image and fill the gaps between the dynamic targets. In actual operation, the structure element is slid on the contour of the binary image, and the structure element is compared with the pixels at the corresponding position in the binary image. If any pixel outside the contour of the structure element matches the pixel value at the corresponding position inside the contour, the pixel outside the contour is also set as a dynamic target, thereby expanding the area of ​​the dynamic target. The number of pixels in the area of ​​the expanded dynamic target is accumulated to obtain the first number of pixels.

[0041] The area of ​​the dynamic target object and the number of first pixels in the area reflect the size and position of the currently captured dynamic target object compared to the test preparation state.

[0042] S140 , performing dynamic region recognition on multiple frames of non-interference and non-jitter test images, and taking a union of the recognized dynamic regions to obtain a reference region.

[0043] Dynamic object capture technology is used to process multiple frames of non-interference and non-jitter test images to obtain dynamic areas and static areas; the dynamic areas in the multiple frames of test images are combined to obtain the reference area.

[0044] Perform the same test process as S110 in advance, and eliminate interference and jitter during the test process. For example, maintain the interference-free state in the test room manually, and use a high-precision anti-shake camera to shoot the test process in real time. Use training samples to train the dynamic object capture image processing model in advance (you can refer to the neural network model or the existing model). The training samples include multiple frames of interference-free and jitter-free images taken during the test, and mark the dynamic targets in the images (including the state changes of the test equipment and dynamic samples). Input multiple frames of interference-free and jitter-free test images into the dynamic object capture image processing model for preprocessing, feature extraction, image registration and fusion operations to identify the dynamic area corresponding to the dynamic target. Take the union of the dynamic areas in each frame of the image to obtain the reference area. The reference area represents the maximum range of the dynamic target's activity, and shields external interference and image jitter, and will be used as a threshold for subsequent judgment.

[0045] S150: Take the intersection of the region of the dynamic target object and the reference region to obtain the proportion of the intersection region in the dynamic target object.

[0046] Take the intersection of the area of ​​the dynamic target in the first difference map / expansion map and the reference area. Assuming that the activity range of the dynamic target in the first difference map / expansion map is within the reference area, the activity range outside the reference area may be caused by interference or jitter. Figure 2 , take the intersection of the dynamic target area and the reference area to obtain the intersection area where the two intersect.

[0047] Then, the number of second pixels in the intersection area and the number of third pixels of the dynamic target are counted. The second pixel number is divided by the third pixel number to obtain the proportion of the intersection area in the dynamic target. For example, if the number of second pixels in the intersection area is 90 and the number of third pixels of the dynamic target is 100, the proportion is 90%.

[0048] S160, calculating the maximum number of pixels of the dynamic target object according to the reference frame and the full image corresponding to the state change of the full test data.

[0049] The difference between the reference frame and the full image is calculated to obtain a second difference map; contour recognition is performed on the second difference map to obtain the maximum number of pixels of the dynamic target object.

[0050] Among them, the state change of the full amount of test data includes the maximum displacement and deformation of the sample, and the display of the test equipment changes. It can be considered that the test process and the state of the sample are controlled to obtain the full amount of images taken when the state of the full amount of test data changes. Similar to the process of obtaining the first difference map, the reference frame and the full amount of images are converted into grayscale images to obtain the full amount of grayscale images and the reference frame grayscale images respectively; the grayscale values ​​of the pixels at the same position in the full amount of grayscale images and the reference frame grayscale images are subtracted to obtain the second difference map. The second difference map reflects the maximum range of change of the target object when the state of the full amount of test data changes. The second difference map is subjected to contour recognition (further expansion processing can be performed), and the number of pixels in the contour is counted to obtain the maximum number of pixels of the dynamic target object. The maximum number of pixels will be used as a threshold for subsequent judgment.

[0051] S170: Determine whether the ratio is greater than a set threshold and whether the first pixel number is less than a maximum pixel number. If yes, execute S171; otherwise, execute S172.

[0052] S171. Keep the target frame image.

[0053] S172: Eliminate the target frame image.

[0054] If the ratio is greater than the set threshold, it means that the dynamic targets detected in the target frame image are basically within the reference area, and it is judged that there is no interference and image jitter, and it needs to be retained. If the ratio is less than or equal to the set threshold, it means that most of the dynamic targets are not within the reference area, which may be caused by interference or jitter and need to be eliminated.

[0055] If the first pixel number is less than the maximum pixel number, it means that the change of the dynamic target is within the maximum change range, and it is judged that there is no interference and image jitter, and it needs to be retained. If the first pixel number is greater than or equal to the maximum pixel number, it means that the change of the dynamic target exceeds the maximum change range, and it is judged that there is interference or image jitter, and it needs to be eliminated.

[0056] S180 , capturing a dynamic target object according to pixel differences between adjacent frames in the retained image, and obtaining a test state displayed by the dynamic target object.

[0057] Assuming that 20 frames of images are retained, determine the area surrounded by the difference pixels between adjacent frames of the 20 frames of images, and the area is the captured dynamic target. If a dynamic target is captured, record the current moment and the test data at the current moment so that it corresponds to the dynamic target. After determining the area of ​​the dynamic target, the test state displayed by it can be obtained through target recognition algorithms, etc. Specifically, color recognition, position recognition and action recognition are performed on the dynamic target, and the color and action of each identified position are recorded as the test state. For example, color recognition is performed based on the pixel color of the dynamic target to identify the color of the indicator light, and the position and action of the sample in the dynamic target are identified based on the shape / color of the sample.

[0058] In an alternative embodiment, see Figure 3 , before calculating the difference between the target frame image and the reference frame in the multiple frames to obtain the first difference map, in response to the user's selection operation in the virtual image, determine the selected area; and intercept the selected area in each frame. The virtual image is an image displayed on the system for the user to select an area, and has the same size as the target frame image. The user can manipulate the mouse to drag a selection box on the virtual image to select an area. This embodiment allows the user to select an area of ​​interest for key monitoring.

[0059] In an optional embodiment, after retaining the target frame image, an adversarial sample set is constructed, and the adversarial sample set includes multiple pairs of positive and negative samples, each pair of positive and negative samples has the same test data, the positive sample is not interfered by the outside world, and the negative sample is interfered by the outside world. The image classification model is trained according to the adversarial sample set so that the image classification model has the ability to classify interference images. Image classification models include but are not limited to random forest models and support vector machine models. Any image in the retained images is input into the image classification model; if it is classified as an interference image, the image is eliminated from the retained images. In this embodiment, on the basis of eliminating interference images by proportion and first pixel value, it is found that there are still some images with interference. In order to further improve the recognition accuracy, the image classification model is used to further eliminate the interference images.

[0060] The present application also provides an electronic device, see Figure 4 , comprising at least one processor 301, and a memory 302 communicatively connected to at least one of the processors 301;

[0061] The memory 302 stores instructions that can be executed by at least one of the processors 301. The instructions are executed by at least one of the processors 301 so that at least one of the processors 301 can execute the above-mentioned machine vision-based test status auxiliary monitoring method, thereby having at least the same advantages as the above-mentioned method.

[0062] Optionally, the electronic device also includes an interface for connecting various components, including a high-speed interface and a low-speed interface. Various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on a memory to display graphical information of a GUI (Graphical User Interface) on an external input / output device (such as a display device coupled to an interface). In other embodiments, if necessary, multiple processors can be used together with multiple memories, and / or multiple buses can be used together with multiple memories. Similarly, multiple electronic devices can be connected (for example, as a server array, a group of blade servers, or a multi-processor system), and each device provides some necessary operations.

[0063] The memory 302 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the machine vision-based test state auxiliary monitoring method in the embodiment of the present application. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 301, that is, realizing the above-mentioned machine vision-based test state auxiliary monitoring method.

[0064] The memory 301 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 302 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0065] The electronic device may further include: an input device 303 and an output device 304. The processor 301, the memory 301, the input device 303 and the output device 304 may be connected via a bus or other means.

[0066] The input device 303 can receive input digital or character information, and the output device 304 can include a display device, an auxiliary lighting device (e.g., LED), and a tactile feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0067] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0068] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A test state auxiliary monitoring method based on machine vision, characterized in that: include: Real-time shooting is performed during the test of automobile samples to obtain multiple frames of images and the corresponding relationship between each frame of image and test data; Calculating the difference between the target frame image and the reference frame in the multiple frames of images to obtain a first difference map; the reference frame corresponds to a test preparation state; The target frame image is any frame image among multiple frame images; Performing contour recognition on the first difference map to obtain an area of ​​the dynamic target and a first number of pixels of the dynamic target; Perform dynamic region recognition on multiple frames of non-interference and non-jitter test images, and take the union of the identified dynamic regions to obtain the reference region; Taking the intersection of the area of ​​the dynamic target object and the reference area to obtain the proportion of the intersection area in the dynamic target object; Calculate the maximum number of pixels of the dynamic target according to the full image corresponding to the state change of the reference frame and the full test data; If the ratio is greater than a set threshold and the first number of pixels is less than the maximum number of pixels, retain the target frame image; The dynamic target is captured according to the pixel difference between adjacent frames in the retained image, and the test state displayed by the dynamic target is obtained.

2. The method according to claim 1, characterized in that: Before calculating the difference between the target frame image and the reference frame in the multiple frames of images to obtain the first difference map, the method further includes: In response to a selection operation of the user in the virtual image, determining a selected area; The selected area is captured in each frame of the image.

3. The method according to claim 2, characterized in that Calculating the difference between the target frame image and the reference frame in the multiple frames of images to obtain a first difference map includes: Converting the target frame image and the reference frame into grayscale images to obtain a target frame grayscale image and a reference frame grayscale image respectively; The grayscale values ​​of pixels at the same position in the target frame grayscale image and the reference frame grayscale image are respectively subtracted to obtain a first difference image.

4. The method according to claim 3, characterized in that Performing contour recognition on the difference map to obtain the area of ​​the dynamic target and the first pixel number of the dynamic target includes: Converting the first difference map into a binary map; Performing contour recognition on the binary image to obtain the region of the dynamic target object; The binary image is dilated to expand the region of the dynamic target object, and the region of the dynamic target object and the first pixel number of the dynamic target object in the dilated image are obtained.

5. The method according to claim 4, characterized in that Perform dynamic region recognition on multiple frames of non-interference and non-jitter test images, and take the union of the identified dynamic regions to obtain the reference region, including: The dynamic object capture technology is used to process multiple frames of non-interference and non-jitter test images to obtain dynamic areas and static areas; The dynamic regions in multiple test images are combined to obtain the reference region.

6. The method according to claim 5, characterized in that Taking the intersection of the area of ​​the dynamic target and the reference area to obtain the proportion of the intersection area in the dynamic target includes: Taking the intersection of the region of the dynamic target object and the reference region to obtain an intersection region; Counting the number of second pixels in the intersection area and the number of third pixels of the dynamic target object; The second number of pixels is divided by the third number of pixels to obtain the proportion of the intersection area in the dynamic target.

7. The method according to claim 6, characterized in that According to the full image corresponding to the state change of the reference frame and the full test data, the maximum number of pixels of the dynamic target is calculated, including: Calculating the difference between the reference frame and the full image to obtain a second difference map; Perform contour recognition on the second difference map to obtain the maximum number of pixels of the dynamic target object.

8. The method according to claim 7, characterized in that After retaining the target frame image, the method further includes: Constructing an adversarial sample set, wherein the adversarial sample set includes multiple pairs of positive and negative samples, each pair of positive and negative samples has the same test data, the positive samples are not interfered by the outside world, and the negative samples are interfered by the outside world; Training an image classification model according to the adversarial sample set so that the image classification model has the ability to classify interference images; Inputting any one of the retained images into the image classification model; If it is classified as a distracting image, the image is removed from the retained images.

9. The method according to claim 8, characterized in that The dynamic target is captured according to the pixel difference between adjacent frames in the retained image, and the test state displayed by the dynamic target is obtained, including: If a dynamic target is captured, the current moment is recorded, and the test data at the current moment is recorded; The dynamic target is subjected to color recognition, position recognition and action recognition, and the color and action of each position recognized are recorded as a test state.

10. An electronic device, characterized in that: include: at least one processor, and a memory communicatively coupled to at least one of the processors; Wherein, the memory stores instructions that can be executed by at least one of the processors, and the instructions are executed by at least one of the processors so that at least one of the processors can execute the machine vision-based test status auxiliary monitoring method described in any one of claims 1-9.

Images