Inspection device, inspection method, and program

Abstract

This contributes to providing inspection equipment, inspection methods, and programs that can perform highly accurate inspections of light sources. [Solution] The system includes a processing unit that identifies the position of a light source in an image obtained by photographing an object including a light source, generates measurement information indicating the temporal change of the light source from the position of the light source in the image, and determines the quality of the light source by comparing the measurement information with reference information indicating the temporal change of the light source in a normal scenario, and a display unit that outputs the result of the determination by the processing unit.

Description

Technical Field

[0001] The present disclosure relates to an inspection apparatus, an inspection method, and a program.

Background Art

[0002] In an instrument panel of an automobile, a number of light sources such as a direction indicator, a seat belt warning light, and an engine warning light are arranged. These light sources are turned on, off, or blink at a specific timing in conjunction with operations such as a key operation by a user of the automobile.

[0003] Conventionally, inspection of these light sources has been performed by a skilled inspector visually or by using a fixed dedicated camera system to statically determine individual light sources.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the inspection by an inspector visually, a highly accurate inspection cannot be performed sufficiently.

[0006] Non-limiting embodiments of the present disclosure contribute to providing an inspection apparatus, an inspection method, and a program capable of performing a highly accurate inspection of a light source.

Means for Solving the Problems

[0007] An inspection apparatus according to one embodiment of the present disclosure includes a processing unit that identifies the position of a light source in an image obtained by photographing an object including a light source, generates measurement information indicating the temporal change of the light source from the position of the light source in the image, and determines the quality of the light source by comparing the measurement information with reference information indicating the temporal change of the light source in a normal scenario, and a display unit that outputs the result of the determination in the processing unit.

[0008] An inspection method according to one embodiment of the present disclosure involves an inspection device that identifies the position of a light source in an image obtained by photographing an object including the light source, generates measurement information indicating the temporal change of the light source from the position of the light source in the image, determines the quality of the light source by comparing the measurement information with reference information indicating the temporal change of the light source in a normal scenario, and outputs the result of the determination.

[0009] A program according to one embodiment of the present disclosure causes an inspection device to perform the following processes: identify the position of the light source in an image obtained by photographing an object including the light source; generate measurement information indicating the temporal change of the light source from the position of the light source in the image; determine the quality of the light source by comparing the measurement information with reference information indicating the temporal change of the light source in a normal scenario; and output the result of the determination.

[0010] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0011] Non-limiting embodiments of this disclosure can perform highly accurate inspection of light sources.

[0012] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features. [Brief explanation of the drawing]

[0013] [Figure 1] Block diagram showing an example configuration of an inspection system according to one embodiment. [Figure 2] A diagram showing an example of the subjects of the inspection. [Figure 3] Figure showing the first example of video footage of an instrument panel. [Figure 4] Figure showing a second example of video footage of an instrument panel. [Figure 5] This diagram shows a third example of video footage of an instrument panel. [Figure 6] Figure showing the first example of a measurement waveform of the brightness of a light source. [Figure 7] Figure showing a second example of the measurement waveform of the light source's brightness. [Figure 8A] A flowchart showing an example of the inspection process flow in one embodiment. [Figure 8B] A flowchart showing an example of the inspection process flow in one embodiment. [Modes for carrying out the invention]

[0014] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same function are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0015] (One embodiment) <Knowledge leading to this disclosure> In the instrument panel of an automobile, a number of light sources such as direction indicators, seat belt warning lights, and engine warning lights are arranged. These light sources turn on, turn off, or blink at specific timings in conjunction with operations such as the key operation of the automobile user.

[0016] Conventionally, for the inspection of these light sources, a method has been used in which a skilled inspector visually checks them, or an individual light source is statically determined using a fixed dedicated camera system.

[0017] However, in the inspection by an inspector's visual observation, a highly accurate inspection cannot be sufficiently performed. For example, in the visual inspection, variations in determination occur depending on the skill level and physical condition of the inspector, and it is difficult to ensure stable quality. Also, in a fixed dedicated camera system, it is difficult to evaluate the "dynamic sequence quality" of whether a plurality of light sources correctly respond in time series to external stimuli such as key operations, and the equipment is expensive and the installation location is limited.

[0018] In recent years, the instrument panel of an automobile has evolved from the conventional display using individual LEDs to an integrated digital meter using a liquid crystal display or an organic EL display. In these display-type instruments, speed displays, warning icons, navigation information, etc. are dynamically drawn on a single screen. Even in such display-type instruments, it is desirable to inspect whether the startup sequence linked to the key operation is correctly executed, and whether the warning display is shown with the intended brightness, color, and blinking pattern.

[0019] Also, for a long time, light sources such as incandescent bulbs and halogen lamps have been used for display devices of automobiles and industrial equipment. Even now, there is a demand for the inspection of these light sources such as incandescent bulbs and halogen lamps in the maintenance market of old vehicles, the restoration of classic cars, old equipment in factories, ships, and railway vehicles.

[0020] This disclosure provides an inspection system including an inspection device that allows anyone, anywhere, to quantitatively and accurately evaluate the quality of complex lighting, flashing, and other scenarios involving multiple light sources, which are triggered by external actions such as key operation for starting a vehicle engine.

[0021] For example, in this embodiment, the effects of hand tremors when an inspector holds an inspection device (e.g., a terminal) in their hand to take a picture are suppressed by combining an inertial sensor built into the terminal with image processing, and the rapid flashing of a light source can also be accurately detected and evaluated.

[0022] <Example of system configuration> Figure 1 is a block diagram showing an example configuration of the inspection system 1 according to this embodiment. The inspection system 1 includes a terminal 10 and an external input device 30. The inspection system 1, as an example, performs an inspection to determine whether the operation of the object to be inspected 20 is normal.

[0023] The object to be inspected 20 is, for example, a light source installed on the instrument panel of a vehicle. The object to be inspected 20 includes one or more light sources. The light source is, for example, at least one of LED (Light Emitting Diode), incandescent bulb, LCD (Liquid Crystal Display), and OLED (Organic Light Emitting Diode).

[0024] The external input device 30 outputs a signal that triggers the timing of the inspection. The external input device 30 is at least one of the following: a wearable sensor 301, an acoustic sensor 302, and a vehicle diagnostic port 303.

[0025] Terminal 10 may be a portable device such as a tablet or a smartphone. Terminal 10 may be, for example, a commercially available tablet device. Terminal 10 includes an imaging unit 101 (camera), a sensor 102, a processor, a recording medium such as memory, a touch panel, and a wireless communication function.

[0026] As shown in Figure 1, terminal 10 includes an imaging unit 101, a sensor 102, a stabilization processing unit 103, a trigger detection unit 104, a waveform generation unit 105, a storage unit 106, a determination unit 107, a display unit 108, and an operation unit 109. The stabilization processing unit 103, the trigger detection unit 104, the waveform generation unit 105, and the determination unit 107 are included in the processing unit 120. The processing unit 120 corresponds to a processor. The storage unit 106 corresponds to a recording medium such as memory. The display unit 108 and the operation unit 109 correspond to a touch panel. The processing unit 120 may communicate with an external device (for example, an external input device 30) using a wireless communication function.

[0027] The imaging unit 101 photographs the object to be inspected 20 and outputs the captured image to the stabilization processing unit 103.

[0028] Sensor 102 includes, for example, an inertial sensor 102a, a gyroscope sensor 102b, and an accelerometer sensor 102c, which are built into the terminal 10. Sensor 102 outputs information regarding the attitude, amount of movement, direction of movement, etc., of the terminal 10 to the stabilization processing unit 103. Information regarding the attitude, amount of movement, direction of movement, etc., of the terminal 10 is referred to as terminal attitude information.

[0029] The stabilization processing unit 103 performs stabilization processing on the video captured by the imaging unit 101. The stabilization processing unit 103 acquires terminal orientation information from the sensor 102 in real time. For example, the stabilization processing unit 103 acquires terminal orientation information from the sensor 102 in synchronization with the video. The stabilization processing unit 103 also detects static feature points within the captured video frame and tracks the amount of movement of these static feature points. By integrating the terminal orientation information and static feature points (sensor fusion), the system corrects video shake caused by camera shake with high precision and generates stable video similar to that captured when the camera is fixed.

[0030] Specifically, the stabilization process includes two processes: the first correction process and the second correction process.

[0031] In the first correction process, the stabilization processing unit 103 acquires the roll angle, pitch angle, and yaw angle of the terminal 10 from the gyro sensor 102b at a sampling rate of 1000 Hz. From these angles, the stabilization processing unit 103 calculates how much the terminal 10 is tilted relative to the horizontal reference and applies an inverse rotation transformation matrix to each video frame of the captured image. As a result, the stabilization processing unit 103 corrects the image so that it always remains horizontal even if the terminal 10 is tilted.

[0032] In the second correction process, the stabilization processing unit 103 extracts static feature points from within the video frame using a feature point detection algorithm. Here, static feature points are, for example, points corresponding to non-flashing parts. For example, static feature points may be the outer frame of the instrument panel that is the object of inspection 20, the outer frame of the meter part, or the numbers, scales, needles, or screws fixed to the meter part. The feature point detection algorithm is at least one of the following, for example, SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), ORB (Oriented FAST (Features from Accelerated Segment Test) and Rotated BRIEF (Binary Robust Independent Elementary Features)). The stabilization processing unit 103 calculates how much and in which direction these feature points have moved between consecutive frames. The stabilization processing unit 103 refers to the translational movement amount of the terminal 10 obtained from the acceleration sensor 102c to supplement the accuracy of feature point tracking. The stabilization processing unit 103 cancels the forward, backward, left, and right movements of the user's hand holding the terminal 10 in each video frame by applying the inverse vector of the calculated movement amount to each video frame.

[0033] The stabilization processing unit 103 cancels camera shake during shooting by performing a first correction process and a second correction process, generates a stable image, and outputs the generated image to the waveform generation unit 105. The generated image is sometimes referred to as a stabilized image.

[0034] The trigger detection unit 104 detects a trigger that indicates the timing of an operation that starts the inspection scenario. Here, an operation that starts the scenario is, for example, turning the key to IGN to prepare to start the engine, or pressing the brake pedal. The trigger detection unit 104 detects information regarding the timing of the inspection based on the output of at least one of the wearable sensor 301, the acoustic sensor 302, and the vehicle diagnostic port 303. Information regarding the timing of the inspection is, for example, a trigger signal that indicates the start of the inspection (the starting point of the inspection scenario). Three methods for acquiring the trigger signal are described below.

[0035] (Method 1: Wearable sensor 301) The wearable sensor 301 is a small accelerometer and gyroscope sensor embedded in the glove of the inspector performing the inspection. The wearable sensor 301 connects to the trigger detection unit 104 of the terminal 10 via short-range wireless communication such as Bluetooth®. The wearable sensor 301 outputs information about the inspector's hand movements (e.g., hand speed, angular velocity) to the trigger detection unit 104. The trigger detection unit 104 recognizes the specific action pattern of the inspector turning the vehicle key using a machine learning model and generates a trigger signal at the timing when the inspector turns the vehicle key. The specific action pattern of the inspector turning the vehicle key is detected by the change in angular velocity detected by the wearable sensor 301.

[0036] (Method 2: Acoustic sensor 302) The acoustic sensor 302 is, for example, a microphone built into the terminal 10. The acoustic sensor 302 outputs information about the sound it has acquired to the trigger detection unit 104. The trigger detection unit 104 captures the mechanical click sound, such as a "click," that occurs when an inspector turns the vehicle's key. The trigger detection unit 104 matches the sound with a pre-learned acoustic pattern and outputs a trigger signal when the degree of agreement between the acoustic pattern of the mechanical click sound and the acquired sound exceeds a threshold.

[0037] (Method 3: Vehicle data linkage) The vehicle diagnostic port 303 is a port that connects the terminal 10 to the vehicle's CAN (Controller Area Network). For example, the inspector connects the terminal 10 to the vehicle's CAN bus via a Bluetooth OBD-II adapter. The vehicle diagnostic port 303 outputs the key position status signal from the CAN to the trigger detection unit 104. The trigger detection unit 104 directly acquires the change in the key position status signal. The change in the key position status signal corresponds to, for example, the change between the key position OFF, ACC (Accessory), and IGN (Ignition). The trigger detection unit 104 may output a trigger signal when the key position becomes IGN, or when the key position becomes ACC, or at both of these timings. In method 3, since the key position status signal can be acquired directly, the timing of when the key position becomes IGN, when the key position becomes ACC, etc. can be acquired with high accuracy.

[0038] The trigger signal may be acquired by at least one of the methods 1 to 3 described above, or by a combination of two or more methods. Furthermore, the trigger signal may be acquired by a method different from those described above.

[0039] The waveform generation unit 105 extracts the time-dependent changes in the brightness of each light source from the stabilized video as a "measured waveform," starting from the timing when the trigger signal is detected (for example, T=0). The waveform generation unit 105 accurately captures the rapid blinking of light sources by combining high frame rate shooting (for example, 120fps or higher) with mathematical interpolation processing such as spline interpolation. The timing for increasing the frame rate of shooting may be after the trigger signal is detected or before the trigger signal is detected. For example, if the blinking of the lamp on the inspection target 20 is used as the trigger, the frame rate may be increased at the timing when the trigger signal is detected. Also, the frame rate of shooting does not have to be changed. If the frame rate of shooting is not changed, the imaging unit 101 shoots the inspection target 20 at the same frame rate regardless of whether or not a trigger signal is detected and the timing of the trigger signal detection, and outputs the captured video to the stabilization processing unit 103.

[0040] For example, the waveform generation unit 105 performs the following processes (1) to (4) to generate a measurement waveform.

[0041] (1) Setting the ROI (Region of Interest) of the light source: The waveform generation unit 105 identifies the position of each light source on the screen from pre-registered vehicle information and sets the ROI. For example, the waveform generation unit 105 identifies the position of static feature points on the screen and identifies the position of each light source on the screen based on the positional relationship between the static feature points and the light sources. The positional relationship between the static feature points and the light sources is stored in association with pre-registered vehicle information.

[0042] (2) Brightness extraction: The waveform generation unit 105 calculates the average luminance value within the ROI for each video frame of the stabilized video. In the case of a color light source, the waveform generation unit 105 individually extracts the luminance of each RGB channel.

[0043] (3) High frame rate shooting: The waveform generation unit 105 drives the imaging unit 101 at a high frame rate (for example, 120fps or 240fps) and acquires video captured at a high frame rate.

[0044] (4) Interpolation process: The waveform generation unit 105 applies cubic spline interpolation to the obtained discrete luminance data points to generate a smooth, continuous waveform. This achieves effective high temporal resolution that exceeds the limitations of the sampling theorem.

[0045] The waveform generation unit 105 outputs the measurement waveform after interpolation processing to the determination unit 107. The measurement waveform is an example of measurement information that shows the time change of the light source. The time change of the light source may be represented in a form other than a waveform, such as a numerical sequence of the light source's brightness value or a multidimensional vector.

[0046] The memory unit 106 is a database that stores reference waveforms and evaluation profiles. The reference waveform is extracted in advance from video footage of the instrument panel's light source operating normally. The reference waveform is the waveform of a normal lighting scenario and is extracted and registered in advance for each vehicle model. The reference waveform is an example of reference information that shows the time change of the light source in a normal scenario. The time change of the light source in a normal scenario may be represented in a form other than a waveform, such as a numerical sequence of light source luminance values ​​or a multidimensional vector.

[0047] The determination unit 107 makes a determination regarding the quality of the light source based on the measured waveform and the reference waveform.

[0048] The determination unit 107 extracts characteristic quantities such as the flashing period, duty cycle, period jitter (standard deviation), rise time, and peak brightness from the measured waveform.

[0049] The determination unit 107 converts the extracted feature quantities of the measurement waveform into a feature quantity vector of the measurement waveform.

[0050] The determination unit 107 extracts features such as the blinking period, duty cycle, period jitter (standard deviation), rise time, and peak brightness from the reference waveform, similar to the feature vector of the measured waveform, and converts the extracted features of the reference waveform into a feature vector of the reference waveform.

[0051] The determination unit 107 calculates the Euclidean distance between the feature vector of the measured waveform and the feature vector of the reference waveform. The determination unit 107 determines that the test target 20 is functioning normally if the Euclidean distance is within the acceptable range set in the evaluation profile. The determination unit 107 determines that the test target 20 is not functioning normally if the Euclidean distance is outside the acceptable range set in the evaluation profile. The case in which the test target 20 is functioning normally is when the test result of the test target 20 is a pass, and the case in which the test target 20 is not functioning normally is when the test result of the test target 20 is a fail.

[0052] If the inspection result for the 20 items to be inspected is unsatisfactory, the judgment unit 107 analyzes which features deviate from the standard and identifies the reason for the failure. The judgment unit 107 generates a diagnostic message indicating the reason for the failure (for example, "Reason for failure: Flashing cycle is 15% slower than specified (possible abnormality in the drive circuit)").

[0053] The display unit 108 is, for example, the display of terminal 10. The operation unit 109 (for example, a touch panel) is superimposed on the display unit 108. The display unit 108 displays the inspection results. For example, if the inspection result is a failure, text indicating the reason for the failure and a button to determine the validity of the failure result are displayed. Also, if the inspection result is a failure, the display unit 108 automatically plays a few seconds of video before and after the moment the failure was determined. The buttons to determine the validity of the failure result include a "Confirm" button, which is pressed when the inspection result is valid, and a "Change" button, which is pressed when the inspection result is not valid and the inspection result should be changed. The inspector checks the played-back video and presses the "Confirm" button if they determine that the inspection result is valid. The inspector checks the played-back video and presses the "Change" button if they determine that the inspection result is not valid (for example, a false positive). When the "Change" button is pressed, the inspection result corresponding to the played-back video is changed from a failure to a pass. Furthermore, if an inspector manually alters the inspection results, the history of the changes is recorded and used to retrain the AI ​​(Artificial Intelligence) model.

[0054] <Examples of subjects to be tested and the testing procedure> Next, we will explain an example of the inspection process for the subject of inspection and the video footage of the subject.

[0055] Figure 2 shows an example of an inspection target 20. Figure 2 shows an example of light sources installed on the instrument panel of a vehicle, which is an example of an inspection target 20. Figure 2 shows an instrument panel including seven light sources A to G, and instruments such as a speedometer and tachometer. Note that the instrument panel shown in Figure 2 and subsequent figures is an example, and some displays have been simplified. The instrument panel shown in Figure 2 and subsequent figures differs from those installed in actual vehicles. Hereafter, inspection target 20 will be referred to as light sources A to G.

[0056] Terminal 10 performs stabilization processing on the video footage of the subject 20 being inspected. In the stabilization processing, the stabilization processing unit 103 extracts static feature points from within the video frame using a feature point detection algorithm.

[0057] Figure 3 shows a first example of video footage of an instrument panel. Figure 3 shows static feature points P extracted using a feature point detection algorithm from the video frames of the instrument panel shown in Figure 2. In the example in Figure 3, multiple static feature points P are extracted from the scale portion of the tachometer M and the needle portion of the tachometer M.

[0058] The stabilization processing unit 103 cancels the forward, backward, left, and right movements of the user's hand holding the terminal 10 in each video frame based on these static feature points between consecutive frames. The waveform generation unit 105, upon detecting a trigger signal, identifies the position of each light source on the screen based on the positional relationship between the static feature points and the light source. The waveform generation unit 105 then calculates the average brightness value within the ROI for each video frame of the stabilized video.

[0059] Figure 4 shows a second example of video footage of the instrument panel. In Figure 4, the on-screen positions of each light source are identified in the video frame of the instrument panel shown in Figure 2, and light sources A to G are lit after receiving a trigger signal.

[0060] Figure 5 shows a third example of video footage of the instrument panel. Figure 5 shows a state where light sources A to C remain lit, while light sources D to G are turned off, at a time later than when the video frames shown in Figure 4 were captured.

[0061] For example, if we represent the measurement time as T and the timing of receiving the trigger signal as T=0, then Figure 4 is an image of the state T=T1 (where T1 is a real number greater than 0), and Figure 5 is an image of the state T=T2 (where T2 is a real number greater than T1). In Figures 4 and 5, the region surrounding light sources A to G corresponds to the ROI.

[0062] The waveform generation unit 105 calculates the average brightness value within the ROI for each video frame of the stabilized video. For example, in the examples in Figures 4 and 5, in the video frame at timing T=T1, light sources A to G show higher brightness values ​​than when they are off, and in the video frame at timing T=T2, light sources A to C show higher brightness values ​​than when they are off, while light sources D to G show the off state.

[0063] The determination unit 107 then compares the measured waveform generated by the waveform generation unit 105 with the stored reference waveform to determine whether the inspection result is a pass or a fail.

[0064] Figure 6 shows a first example of the measurement waveform of the brightness of a light source. Figure 6 illustrates the measurement waveforms of light sources A, B, and D shown in Figures 2-5, overlaid with a reference waveform. For example, as shown in Figures 4 and 5, the brightness of light sources A and B is high at T=T1 and T=T2, while the brightness of light source D is high at T=T1 and the same as when the light is off at T=T1.

[0065] In the example shown in Figure 6, the similarity (matching degree) between the measurement waveform of light source A and the reference waveform is above a predetermined level. Similarly, the similarity (matching degree) between the measurement waveforms of light sources B and C and the reference waveform is also above a predetermined level. Therefore, in the example shown in Figure 6, the determination unit 107 determines that the inspection result is acceptable.

[0066] Figure 7 shows a second example of the measurement waveform of the light source brightness. Illustratively, as in Figure 6, Figure 7 displays the measurement waveforms of light sources A, B, and D shown in Figures 2-5, overlaid with a reference waveform. For example, as shown in Figures 4 and 5, the brightness of light sources A and B is high at T=T1 and T=T2, while the brightness of light source D is high at T=T1 and the same brightness as when the light is off at T=T1.

[0067] In the example shown in Figure 7, the similarity (match) between the measured waveform of light source A and the reference waveform is above a predetermined level. On the other hand, the timing at which light source B turns off is earlier than the timing shown in the reference waveform (T=T3), and the brightness of light source D is lower than the brightness shown in the reference waveform. Therefore, the similarity (match) between the measured waveforms of light sources B and C and the reference waveform is below a predetermined level, and in the example shown in Figure 7, the judgment unit 107 determines that the inspection result is unsatisfactory.

[0068] In Figures 6 and 7, the determination is made based on the difference between the measured waveform, which shows the change in brightness over time, and the reference waveform, but this disclosure is not limited to this. As described above, the determination unit 107 may extract feature quantities such as the blinking period, duty cycle, period jitter (standard deviation), rise time, and peak brightness from the measured waveform, convert the extracted feature quantities of the measured waveform into a feature quantity vector of the measured waveform, and make a determination by comparing the measured waveform and the reference waveform in the form of a feature vector.

[0069] <Inspection process flow in this embodiment> Figures 8A and 8B are flowcharts illustrating an example of the inspection process flow in this embodiment. Note that the "A" after S114 in Figure 8A connects to the "A" before S115 in Figure 8B. The flow shown in Figures 8A and 8B is initiated, for example, when an inspector activates terminal 10 to perform an inspection.

[0070] On terminal 10, the system is initialized by the inspector's operation, the inspection application is launched, and the sensor 102 is activated (S101).

[0071] At terminal 10, the inspector performs the inspection settings (S102). For example, the inspection settings involve selecting the vehicle types to be inspected (20 vehicles). When the determination unit 107 of terminal 10 receives an operation to select a vehicle type, it obtains the evaluation profile corresponding to the selected vehicle type and the reference waveform data corresponding to the selected vehicle type from the storage unit 106.

[0072] The imaging unit 101 of terminal 10 is activated, and the inspector points the imaging unit 101 at the inspection target 20, causing the imaging unit 101 to capture an image of the inspection target 20 (S103). For example, the imaging unit 101 captures the image at a speed of 120fps.

[0073] The stabilization processing unit 103 starts stabilizing the captured video (S104). Stabilization may be performed continuously on the captured video. Alternatively, stabilization may be performed in real time on the captured video.

[0074] The stabilization processing unit 103 determines whether the captured video has become stable through stabilization processing (S105). For example, the stabilization processing unit 103 estimates the magnitude of the video distortion from the amount of movement of the static feature points of the video, and determines that the video is in a stable state if the video distortion is suppressed to below a predetermined level.

[0075] If the system is not stable (NO in S105), the stabilization processing unit 103 performs further adjustments by stabilizing the captured video (S106). Then, the flow returns to S105.

[0076] When a stable state is reached (YES in S105), the display unit 108 displays a message instructing the start of the inspection (S107). The message prompting the start of the inspection may be, for example, "Turn the key to IGN." In S107, the trigger detection unit 104 waits for a trigger.

[0077] The trigger detection unit 104 determines whether or not an external trigger has been detected (S108).

[0078] If no external trigger is detected (NO in S108), the flow returns to S108, and the trigger detection unit 104 waits for a trigger.

[0079] If an external trigger is detected (YES in S108), the trigger detection unit 104 confirms the trigger signal, sets the starting point of the inspection sequence (for example, time T=0), and starts the inspection sequence (S109).

[0080] The imaging unit 101 starts high-speed continuous shooting (S110).

[0081] The waveform generation unit 105 identifies and tracks the ROI in the stabilized video, which has been captured at high speed and stabilized (S111). There may be one ROI or multiple ROIs.

[0082] The waveform generation unit 105 extracts luminance (S112). The luminance may be extracted for each RGB channel. The extracted luminance may be extracted as a change in HSV space.

[0083] The waveform generation unit 105 determines whether a predetermined measurement time has elapsed (S113). For example, the measurement time is about 10 seconds.

[0084] If the measurement time has not elapsed (NO in S113), the flow returns to S111.

[0085] If the measurement time has elapsed (YES in S113), the waveform generation unit 105 performs waveform interpolation processing (S114). The waveform generation unit 105 performs waveform interpolation processing such as spline interpolation and outputs the measured waveform to the determination unit 107.

[0086] The determination unit 107 extracts feature quantities from the measured waveform (S115). For example, at least one of the following features is extracted: blinking period, duty cycle, jitter, rise time, peak brightness, etc. A feature quantity vector of the measured waveform is generated from the extracted feature quantities of the measured waveform.

[0087] The determination unit 107 performs a comparison process between the measured waveform and the reference waveform (S116). Through the comparison process (e.g., correlation process), the distance between the feature vector of the measured waveform and the feature vector of the reference waveform is calculated. The calculated distance indicates the similarity (or degree of agreement) between the measured waveform and the reference waveform. For example, the smaller the calculated distance, the higher the similarity between the measured waveform and the reference waveform.

[0088] The determination unit 107 determines whether the calculated distance is within the acceptable range (S117).

[0089] If the calculated distance is within the acceptable range (YES in S117), the test result is determined to be a pass (S118).

[0090] The determination unit 107 then records the inspection result, indicating that the inspection result is a pass, and the data of the object that was determined to pass (for example, the captured video) (S129). The inspection flow then ends.

[0091] If the calculated distance is outside the acceptable range (NO in S117), the judgment unit 107 determines that the inspection result is unsatisfactory and identifies the reason for the unsatisfactory result (e.g., NG) (S119). For example, a diagnostic report is generated as the reason for NG, showing the results of diagnosing abnormalities in the period, abnormalities in brightness, response delays, etc.

[0092] The determination unit 107 extracts the video footage of the object that has been determined to be unsatisfactory (S120). For example, if a brightness anomaly occurs, the video footage from a specific time before and after the timing of the brightness anomaly is extracted.

[0093] The display unit 108, under the control of the judgment unit 107, displays a message indicating that the product has failed (S121). When displaying a message indicating failure, the video of the product that has been judged to have failed may be automatically played, or the video may be played in slow motion. The inspector checks the displayed video and decides whether or not to confirm that the inspection result of the light source in question has failed. The inspector inputs the result of their decision via the operation unit 109.

[0094] The determination unit 107 receives a judgment from the inspector via the operation unit 109 (S122).

[0095] The determination unit 107 determines whether the inspector's judgment is final or not (S123). For example, it determines whether the inspector has entered a confirmation via the operation unit 109 that the terminal 10's judgment that the inspection result is a failure is final.

[0096] If the inspector's judgment is confirmed (YES in S123), the judgment that the inspection result is a failure is confirmed (S124).

[0097] The judgment unit 107 then records the inspection result of "failure" and the data (e.g., video) of the object that was judged to be failure (S125). The inspection flow then ends.

[0098] If the inspector's judgment is not final (NO in S123), for example, if the inspector requests that a failing inspection result be changed to a passing result, the judgment unit 107 accepts the operation to change the inspection result to a passing result and changes the inspection result to a passing result (S126).

[0099] The judgment unit 107 approves the revised test result as a pass and adds the data (e.g., video) containing the approved test result to the training data (S127).

[0100] The determination unit 107 then records the test result, indicating that the test result is a pass, and the data of the object that was determined to pass (S128). The test flow then ends.

[0101] <Variation 1: Application to display-type instruments> In the embodiments described above, an instrument panel equipped with meters and individual light sources (e.g., LEDs) was used as an example, but this disclosure is not limited thereto. This disclosure may also be applied to the inspection of integrated instruments using liquid crystal displays or organic EL displays, not just individual LEDs. Hereinafter, integrated instruments using liquid crystal displays or organic EL displays will be referred to as display-type instruments.

[0102] (Inspection procedure for display-type instruments) For display-type instruments, the following procedural changes, exemplified in (1) to (4), apply.

[0103] (1)ROI setting The display area to be inspected is pre-registered as coordinates on the screen. Since the vehicle's display-type instrument panel includes multiple display elements, multiple ROIs are set simultaneously. Here, the display area to be inspected includes, for example, the numerical portion of the speedometer, the engine warning icon, the seatbelt warning icon, and so on, at least one of these.

[0104] (2) Detection of display changes For each ROI, not only are the brightness changes over time tracked, but also the changes in hue, saturation, and value in the HSV space. For example, the color change when an engine warning light transitions from "flashing yellow" to "solid red" is accurately captured.

[0105] (3) Comparison of information from the reference pattern with the measured information In the case of display-type instruments, the reference waveform is described not merely as a time series of brightness, but as a "scenario template" that includes time-series events of multiple display elements, such as "the background lights up white at T=0 seconds," "the speedometer needle sweeps from 0 km / h to 180 km / h at T=1 second," and "the seatbelt icon starts flashing yellow at T=2 seconds." In other words, it is not limited to waveforms where brightness changes with time, but information such as changes in brightness, color, and needle position with respect to time is stored as reference pattern information. Furthermore, for the behavior of each measured ROI, patterns such as changes in brightness, color, and needle position with respect to time are measured, and the measured information is compared with the reference information described as a "scenario template." Then, it is determined whether the measured information has a degree of agreement (or similarity) with the reference information that is above a predetermined level.

[0106] (4) Display-specific defect detection Display-type instruments have different failure modes than individual LEDs. In other words, display-type instruments may fail inspection based on conditions different from those for individual LEDs. For example, display delay, color unevenness, and burn-in may be detected through analysis of the measured information. Based on the detected information, the inspection result may be judged as failing. Display delay refers to, for example, a situation where the response speed of an LCD or OLED display is slower than a predetermined time. Color unevenness refers to a situation where the difference between the reference color and the detected color exceeds a predetermined level due to the aging degradation of the LCD or OLED display. Burn-in refers to a situation where, when the display switches, the image from before the switch remains as an afterimage on the display after the switch.

[0107] <Variation 2: Application to light sources other than LEDs> In the embodiments described above, an example was given in which the light source is an LED, but this disclosure is not limited to this. This disclosure may be applied to the inspection of light sources other than LEDs, such as incandescent bulbs and halogen lamps. Herein, as an example, the inspection of a light bulb may be performed as shown in (1) to (4) below.

[0108] (1) Evaluation of startup characteristics An incandescent light bulb takes several hundred milliseconds from the moment power is applied until the filament heats up and reaches a steady brightness. This "rise time," from the start of power application to reaching steady brightness, is evaluated from the measured waveform to determine if it falls within a specified range. If the rise time is longer than the specified range (for example, if the rise is slow), the incandescent light bulb is judged to have failed the inspection. Reasons for failing the incandescent light bulb inspection include filament degradation and insufficient power supply voltage.

[0109] (2) Change in color temperature The light bulb emits a reddish tint during startup and becomes white in a steady state. In this disclosure, the time change in color temperature is measured by tracking the brightness change of each RGB (Red, Green, Blue) channel for each channel. The measured time change in color temperature is then compared with a "reference color temperature curve" that corresponds to the time change of a reference color temperature.

[0110] (3) Brightness stability If the brightness of a light bulb flickers while it is steadily lit, power supply noise or poor contact of the bulb may be suspected. Frequency analysis of the high-frequency components (flicker) of the measurement waveform showing the time change of the light bulb's brightness is performed, and abnormal flickering of the light bulb's brightness is detected. For example, if the detected high-frequency components of the brightness are above a predetermined value, the inspection result may be judged as failing because there is an abnormality in the flickering of the light bulb's brightness.

[0111] (4) Afterglow when the lights are turned off Even after being switched off, a light bulb continues to emit light for a short time due to the residual heat of the filament. This "afterglow time" is extracted from the measured waveform and compared to a standard afterglow time. For example, if the measured afterglow time is longer than the standard afterglow time, it may be determined that there is a problem with the light bulb or the power supply, and the test result is deemed unsatisfactory.

[0112] (The unique characteristics of halogen lamps) Halogen lamps have a faster response time and higher brightness than incandescent bulbs. As described in the above embodiment, by performing high-frame-rate imaging, the steep rise characteristics of halogen lamps can be measured with high accuracy.

[0113] (Examples of application to legacy equipment) In the embodiments described above, an example was given where the object of inspection was the instrument panel of a vehicle, but this disclosure is not limited to this. For example, factory control panels are often designed to illuminate with a red incandescent light bulb when an abnormality occurs. This disclosure shows that the dynamic quality of these conventional light sources can also be quantitatively evaluated using the same methods as modern LEDs.

[0114] <Other Embodiments> This disclosure is not limited to the embodiments described above and may be applied to a variety of display devices. For example, this disclosure can be applied to the inspection of any display device having a light source, such as instrument panels on factory production lines, railway signals, and status lamps on medical devices. Furthermore, while the above embodiments give an example where the external trigger is a key operation, the external trigger is not limited to a key operation, but can be initiated by any operation or external stimulus, such as a button press or switch operation.

[0115] In this disclosure, the term "light source" encompasses all devices that convert electrical energy into light. This includes solid-state elements (LEDs, organic ELs), gas discharge (fluorescent lamps, neon tubes, HID lamps), and thermal radiation (incandescent bulbs, halogen lamps). Regardless of the physical light emission principle of the light source, the core idea of ​​this disclosure—"capturing temporal changes in brightness as a waveform and comparing it to a reference"—is universally applicable.

[0116] According to the embodiment described above, complex sequence inspections that were difficult even for skilled personnel can be automated using commercially available tablets, etc., without the need for expensive fixed jigs or dedicated cameras, and highly accurate inspections can be performed.

[0117] Furthermore, according to this embodiment, by integrating the inertial sensor provided in the terminal 10 with image processing, stable images can be generated even when the inspector holds the terminal 10 without fixing it in place, just as when it is fixed. Therefore, highly accurate inspections can be performed with a simple procedure and at a reduced cost.

[0118] Furthermore, according to this embodiment, by applying a flexible method for detecting external triggers, it is possible to adapt to various inspection environments.

[0119] Furthermore, according to this embodiment, if the inspection result is unsatisfactory, the inspector can manually re-evaluate the product, and the inspection result can be changed based on that re-evaluation. This allows for rapid feedback to the manufacturing process, going beyond simple pass / fail judgments, and accelerating the quality improvement cycle.

[0120] Furthermore, this embodiment is applicable not only to light sources such as LEDs, but also to light sources such as incandescent bulbs and halogen lamps, as well as liquid crystal displays and organic EL displays, thus supporting a wide range of industrial fields.

[0121] Furthermore, according to this embodiment, it is possible to improve and standardize inspection quality and reduce inspection man-hours.

[0122] This disclosure can be implemented using software, hardware, or software integrated with hardware.

[0123] Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0124] The method of integration is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0125] Furthermore, if advancements in semiconductor technology or other derived technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, it would be possible to use those technologies to integrate functional blocks. The application of biotechnology, for example, is a possibility.

[0126] This disclosure is applicable to all types of devices, systems, and equipment having communication capabilities (collectively referred to as communication equipment). Communication equipment may include a radio transceiver and a processing / control circuit. A radio transceiver may include a receiver and a transmitter, or both as functions. A radio transceiver (transmitter, receiver) may include an RF (Radio Frequency) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or similar. Non-exclusive examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned devices.

[0127] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting equipment, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0128] Furthermore, in recent years, Cyber-Physical Systems (CPS), a new concept in IoT (Internet of Things) technology that creates new added value through information linkage between the physical and cyber spaces, has been attracting attention. This CPS concept can also be adopted in the above-described embodiment.

[0129] In other words, as a basic configuration of CPS, for example, edge servers located in physical space and cloud servers located in cyberspace are connected via a network, and processing can be distributed and performed by the processors installed on both servers. Here, it is preferable that each processing data generated on the edge server or cloud server is generated on a standardized platform, and by using such a standardized platform, it is possible to improve efficiency when building systems that include various diverse groups of sensors and IoT application software.

[0130] Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these.

[0131] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0132] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0133] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure.

[0134] The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above. [Industrial applicability]

[0135] One embodiment of the present disclosure is suitable for a system for inspecting an instrument panel. [Explanation of Symbols]

[0136] 1. Inspection System 10 devices 20 subjects of testing 30 External input device 101 Imaging Unit 102 Sensors 102a Inertial Sensor 102b Gyroscope Sensor 102c Accelerometer 103 Stabilization Processing Unit 104 Trigger detection unit 105 Waveform generation section 106 Storage section 107 Judgment section 108 Display section 109 Operation section 120 Processing Unit 301 Wearable Sensor 302 Acoustic Sensor 303 Vehicle Diagnostic Port

Claims

1. The position of the light source in the image obtained by photographing an object including the light source is identified. From the position of the light source in the aforementioned video, measurement information indicating the time change of the light source is generated. A processing unit that determines the quality of the light source by comparing reference information showing the time change of the light source in a normal scenario with the measurement information, A display unit that outputs the result of the determination in the processing unit, An inspection device equipped with the following features.

2. The processing unit identifies the position of the static feature points of the target in the video, and corrects the shaking of the video based on the position of the static feature points and information regarding the posture and movement of the inspection device. The inspection apparatus according to claim 1.

3. The inspection device performs rotational correction to maintain the horizontal orientation of the image based on information regarding the orientation of the inspection device, and corrects the translational movement of the inspection device based on information regarding the tracking of static feature points and the movement of the inspection device. The inspection apparatus according to claim 2.

4. The processing unit starts generating the measurement information when it detects a trigger indicating the start timing of the time change. The inspection apparatus according to claim 1.

5. The trigger is detected based on at least one of the following: a detection result indicating that a person inspecting the object is manipulating the object; sounds generated by the manipulation of the object; and information regarding the state of the object output from the object. The inspection apparatus according to claim 4.

6. The processing unit performs a first display control to display on the display unit the reason why the quality of the light source was determined to be below a predetermined standard, and the measurement information for the time during which the quality was below the predetermined standard. The inspection apparatus according to claim 1.

7. If the processing unit obtains information indicating that the result of the determination should be changed after the first display control, it shall change the result of the determination. The inspection apparatus according to claim 6.

8. The inspection device, This is a portable terminal having an imaging unit for capturing the aforementioned video. The inspection apparatus according to claim 1.

9. The inspection device, The position of the light source in the image obtained by photographing an object including the light source is identified. From the position of the light source in the aforementioned video, measurement information indicating the time change of the light source is generated. The quality of the light source is determined by comparing reference information showing the time variation of the light source in a normal scenario with the measurement information. Output the result of the judgment. Testing method.

10. In the inspection device, The position of the light source in the image obtained by photographing an object including the light source is identified. From the position of the light source in the aforementioned video, measurement information indicating the time change of the light source is generated. The quality of the light source is determined by comparing reference information showing the time variation of the light source in a normal scenario with the measurement information. Output the result of the judgment. A program that executes a process.

Images