Apparatus, method, and electronic device for recognizing a signal of a vehicle direction indicator

Abstract

To provide a recognition device and recognition method of a signal of a direction indicator of a vehicle, and an electronic apparatus.SOLUTION: A method sequentially performs vehicle detection for each frame in a frame sequence including at least one frame period, performs vehicle tracking on the basis of a vehicle detection result of each frame, acquires a vehicle tracking result, performs detection of a lamp on the basis of the vehicle detection result of each frame, acquires a lamp detection result, performs detection of a display direction of a vehicle on the basis of the vehicle detection result of each frame, acquires a vehicle display direction detection result, performs pairing of lamps on the basis of the vehicle tracking result, the lamp detection result and the vehicle display direction detection result, acquires a lamp pairing result, extracts luminance for a location area of the lamp on the basis of the lamp pairing result, acquires a luminance extraction result, performs detection of a signal of a direction indicator of the vehicle on the basis of the luminance extraction result, and acquires a detection result of the signal of the direction indicator of the vehicle.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to the field of information technology.

Background Art

[0002] In recent years, advanced road traffic technologies have developed rapidly and are gradually being applied in various fields. Recognizing the signal of a vehicle's direction indicator (turn lamp) helps to understand the intentions of other road users. For example, it is very useful for reminding a driver of the possibility of danger. Also, regarding the determination of traffic accidents based on in-vehicle cameras, the recognized signal of the vehicle's direction indicator is very important information. Therefore, there is a need to develop a robust machine vision method to achieve this task.

[0003] Currently, most methods for recognizing the signal of a vehicle's direction indicator are based on clustering bright regions and pairing vehicle lamps. This method mainly uses a Gaussian mixture model to cluster bright regions, extracts the characteristics of the light spots of the lamps in each cluster, and determines the correct light spots of the lamps through lamp pairing.

[0004] Note that the above description of the background art is for the purpose of clearly and completely understanding the technical solution of the present invention and is described to enable those skilled in the art to understand. These technical solutions are merely described as the background art part of the present invention and are not well-known to those skilled in the art.

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, according to the discovery of the inventor of the present invention, in the daytime environment, the brightness of the lamp is always weak, and the clustering result of bright areas cannot accurately reflect the existing lamps. Similarly, environmental reflection also has an adverse effect on the extraction of nighttime lamps. Therefore, in actual situations, it is difficult to recognize the signal of the vehicle's direction indicator only by the method based on the clustering of bright areas and the pairing of vehicle lamps, and misrecognition is likely to occur.

[0006] In order to solve at least one of the above problems, embodiments of the present invention provide a recognition device, a recognition method, and an electronic device for the signal of a vehicle's direction indicator, which can recognize the signal of the vehicle's direction indicator in various environments, have a high recognition accuracy, and a wide application range.

Means for Solving the Problems

[0007] In a first aspect of an embodiment of the present invention, there is provided a recognition device for the signal of a vehicle's direction indicator, including: a first detection unit that sequentially performs vehicle detection on each frame in the input frame sequence, where the frame sequence includes at least one frame period; a tracking unit that performs vehicle tracking based on the vehicle detection results of each frame and obtains a vehicle tracking result; a second detection unit that performs lamp detection based on the vehicle detection results of each frame and obtains a lamp detection result; a third detection unit that performs detection of the display direction of the vehicle based on the vehicle detection results of each frame and obtains a vehicle display direction detection result; a matching unit that performs lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result and obtains a lamp pairing result; an extraction unit that extracts the brightness of the lamp location area based on the lamp pairing result and obtains a brightness extraction result; and a fourth detection unit that performs detection of the signal of the vehicle's direction indicator based on the brightness extraction result and obtains a detection result of the vehicle direction indicator signal.

[0008] In a second aspect of an embodiment of the present invention, there is provided an electronic device including the device according to the first aspect of the embodiments of the present invention.

[0009] In a third aspect of an embodiment of the present invention, there is provided a method for recognizing a signal of a direction indicator of a vehicle, the method including steps of sequentially performing vehicle detection for each frame in an input frame sequence, where the frame sequence includes at least one frame period; performing vehicle tracking based on the vehicle detection result of each frame to obtain a vehicle tracking result; performing lamp detection based on the vehicle detection result of each frame to obtain a lamp detection result; performing detection of a display direction of the vehicle based on the vehicle detection result of each frame to obtain a vehicle display direction detection result; performing lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result to obtain a lamp pairing result; extracting luminance for a location area of the lamp based on the lamp pairing result to obtain a luminance extraction result; and performing detection of a signal of the direction indicator of the vehicle based on the luminance extraction result to obtain a detection result of the signal of the vehicle direction indicator.

[0010] Advantageous effects of the present invention are as follows. By performing lamp detection based on a vehicle detection result, that is, by using a detection method instead of a conventional clustering method, the accuracy of lamp detection can be improved. When performing lamp pairing, by considering the combination of the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, the accuracy and efficiency of lamp pairing can be improved. Therefore, the signal of the direction indicator of the vehicle can be recognized in various environments, with a high recognition accuracy and a wide application range.

[0011] Specific embodiments of the present invention are disclosed in detail as shown in the following description and drawings, showing a manner in which the principles of the present invention can be adopted. It should be noted that the embodiments of the present invention are not limited in scope. The embodiments of the present invention include various modifications, corrections, and equivalents within the scope of the gist and content of the appended claims.

[0012] Features described and / or shown in one embodiment may be used in one or more other embodiments in the same or similar manner, combined with features in other embodiments, or may replace features in other embodiments.

[0013] Note that when the term "comprising / including" is used in the present text, it means the presence of features, elements, steps or components, and does not exclude the presence or addition of one or more other features, elements, steps or components.

Brief Description of the Drawings

[0014] The drawings included herein are for understanding the embodiments of the present invention, form a part of this specification, are for illustrating the embodiments of the present invention, and explain the principles of the present invention in combination with the written description. Note that the drawings described herein are only for explaining the embodiments of the present invention, and those skilled in the art can easily obtain other drawings based on these drawings.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0015] The above and other features of the present invention can be understood from the drawings and the following description. In the specification and the drawings, specific embodiments of the present invention, that is, some embodiments that conform to the principles of the present invention, are disclosed. It should be noted that the present invention is not limited to the disclosed embodiments, and the present invention includes all modifications, variations, and equivalents within the scope of the claims.

[0016] <Example 1> The embodiment of the present invention provides a recognition device for a signal of a vehicle direction indicator. FIG. 1 is a schematic diagram of a recognition device for a signal of a vehicle direction indicator according to Embodiment 1 of the present invention.

[0017] As shown in FIG. 1, the recognition device 100 for a signal of a vehicle direction indicator includes a first detection unit 101, a tracking unit 102, a second detection unit 103, a third detection unit 104, a matching unit 105, an extraction unit 106, and a fourth detection unit 107.

[0018] The first detection unit 101 sequentially performs vehicle detection for each frame in the input frame sequence. The frame sequence includes at least one frame period.

[0019] The tracking unit 102 performs vehicle tracking based on the vehicle detection result of each frame and obtains a vehicle tracking result.

[0020] The second detection unit 103 performs lamp detection based on the vehicle detection result of each frame and obtains a lamp detection result.

[0021] The third detection unit 104 performs detection of the display direction of the vehicle based on the vehicle detection result of each frame and obtains a vehicle display direction detection result.

[0022] The matching unit 105 performs lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, and obtains a lamp pairing result.

[0023] The extraction unit 106 extracts the luminance of the lamp's location area based on the lamp pairing result and obtains the luminance extraction result.

[0024] The fourth detection unit 107 detects the signal of the vehicle's direction indicator based on the luminance extraction result and obtains the detection result of the vehicle direction indicator signal.

[0025] In one aspect of an embodiment of the present invention, the input frame sequence may be a video including a vehicle obtained by various methods. For example, it may be a video captured by an in-vehicle camera or a video captured by a road monitoring camera.

[0026] The frame sequence includes at least one frame period (frame duration). The number of frame periods included in the frame sequence and the number of frames included in each frame period may be determined according to actual needs.

[0027] For example, each frame period includes 25 frames, and the frame sequence includes one or more sets of 25 frames.

[0028] In one aspect of an embodiment of the present invention, the first detection unit 101 sequentially performs vehicle detection for each frame in the input frame sequence. In other words, the first detection unit 101 performs vehicle detection for each frame and outputs the vehicle detection result of each frame respectively.

[0029] The vehicle detection result output by the first detection unit 101 is, for example, the detection frame of each detected vehicle and the corresponding position information.

[0030] The first detection unit 101 may detect a vehicle using a model obtained based on a deep learning method. For example, the first detection unit 101 sequentially performs vehicle detection for each frame in the input frame sequence using a Feature Pyramid Networks (FPN). By doing so, more accurate vehicle detection results can be obtained, thus ensuring the accuracy of subsequent lamp detection and pairing.

[0031] In one aspect of an embodiment of the present invention, the tracking unit 102 performs vehicle tracking based on the vehicle detection results of each frame and obtains vehicle tracking results. The tracking unit 102 may perform vehicle tracking using various tracking methods, for example, a network based on Deep Sort. By doing so, good multi-target tracking results can be obtained.

[0032] In one aspect of an embodiment of the present invention, in order to further improve the accuracy of the tracking results, a correction mechanism for the tracking results is designed.

[0033] FIG. 2 is a schematic diagram of one of the tracking units in Embodiment 1 of the present invention. As shown in FIG. 2, the tracking unit 102 includes a comparison unit 201 and a first determination unit 202.

[0034] The comparison unit 201 compares the vehicle detected in the current frame with the vehicles tracked in a plurality of predetermined frames before the current frame, and calculates the overlap ratio of the two compared vehicles.

[0035] When the overlap ratio of the vehicle detected in the current frame and the vehicles tracked in a plurality of predetermined frames before the current frame is greater than a first threshold, the first determination unit 202 uses the vehicle detected in the current frame as the vehicle tracking result of the current frame.

[0036] In one aspect of an embodiment of the present invention, the predetermined number may be determined according to actual requirements. For example, the plurality of predetermined frames before the current frame are the five frames before the current frame.

[0037] For example, when N vehicles are tracked in the five frames before the current frame and M vehicles are detected in the current frame, the comparison unit 201 compares the M vehicles detected in the current frame with the N vehicles tracked in the five frames before the current frame one by one, and calculates the overlapping rate of the two compared vehicles. Here, both N and M are positive integers.

[0038] When calculating the overlapping rate, for example, the overlapping rate of the detection frames of two vehicles may be calculated.

[0039] When the overlapping rate of the vehicle detected in the current frame and the vehicle tracked in a predetermined number of multiple frames before the current frame is greater than the first threshold, the first determination unit 202 uses the vehicle detected in the current frame as the vehicle tracking result of the current frame. For example, when the overlapping rate of the i-th vehicle among the M vehicles currently detected and the j-th vehicle among the N vehicles tracked in the five frames before is greater than the first threshold, the i-th vehicle is used as the vehicle tracking result of the current frame.

[0040] Accordingly, by correcting the vehicle tracking result based on the comparison result between the vehicle detection result of the current frame and the vehicle tracking results of the previous multiple frames, the accuracy of the tracking result can be further improved.

[0041] In one aspect of the embodiment of the present invention, the second detection unit 103 performs lamp detection based on the vehicle detection result of each frame and obtains a lamp detection result. For example, lamp detection is performed within the detection frame of the vehicle detected in each frame.

[0042] The second detection unit 103 may use various detection methods. For example, the second detection unit 103 uses the YOLO V3 network to perform lamp detection. The YOLO V3 network is a lightweight network and has a high recognition accuracy.

[0043] In one aspect of an embodiment of the present invention, the third detection unit 104 performs detection of the display direction of a vehicle using a classifier for the vehicle display direction based on the vehicle detection result of each frame. Here, the classifier for the vehicle display direction is trained by a deep learning method.

[0044] In one aspect of an embodiment of the present invention, the vehicle display direction means the direction of the vehicle indicated by the vehicle detected in the image of each captured frame, that is, the part of the vehicle indicated by the detected vehicle. For example, in the image, when the front part of the vehicle is displayed, the display direction of the vehicle is "front".

[0045] In one aspect of an embodiment of the present invention, the number of classes output by the vehicle display direction classifier may be determined according to actual requirements. For example, the vehicle display direction classifier may output eight classes.

[0046] FIG. 3 is a schematic diagram of one of various vehicle display directions of Embodiment 1 of the present invention. As shown in FIG. 3, in the upper row, from left to right, they are "front", "right front", "left front", and "right", and in the lower row, from left to right, they are "back", "right back", "left back", and "left".

[0047] Thereby, detecting the display direction of the vehicle is useful for specifying the lamp position and pairing the lamps.

[0048] In one aspect of an embodiment of the present invention, it is not limited to the processing order of the tracking unit 102, the second detection unit 103, and the third detection unit 104. For example, the tracking unit 102, the second detection unit 103, and the third detection unit 104 may perform processing in parallel or may perform processing one by one.

[0049] In one aspect of an embodiment of the present invention, after the processing of the tracking unit 102, the second detection unit 103, and the third detection unit 104 is completed, the matching unit 105 performs lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, and obtains a lamp pairing result.

[0050] In one aspect of an embodiment of the present invention, the vehicle tracking result represents the continuously detected result in the frame sequence of the vehicle, and includes, for example, vehicle ID and position change information. The lamp detection result represents the lamp area further detected in the detection frame of the vehicle. The vehicle display direction detection result represents the direction displayed in the image of the vehicle.

[0051] For example, in the case of a specific vehicle in the vehicle tracking result of the current frame, as shown in the third image from the left at the bottom of FIG. 3, the display direction of the vehicle is "left back", and if two lamp areas are further detected in the detection frame of the vehicle, based on the "left back" which is the display direction of the vehicle, it may be determined that these two lamp areas represent the rear lamp (rear turn lamp) on the left side and the rear lamp on the right side of the vehicle. Then, pair these two lamp areas as a pair of rear lamps. Thereby, the pairing of the lamps can be performed accurately and efficiently.

[0052] In one aspect of an embodiment of the present invention, the extraction unit 106 extracts the luminance of the lamp location area based on the lamp pairing result, and obtains a luminance extraction result. The extraction unit 106 may extract the luminance using various methods.

[0053] For example, convert the image format from RGB to HSL, extract the luminance component from the HSL format image, and perform binarization processing on the image with the luminance component extracted based on a predetermined threshold value to binarize the image.

[0054] The predetermined threshold value may be determined according to actual requirements. For example, in a daytime scene, the predetermined threshold value is 127, and in a nighttime scene, the predetermined threshold value is 220.

[0055] In one aspect of an embodiment of the present invention, the fourth detection unit 107 detects a signal of the vehicle's direction indicator based on the luminance extraction result and obtains the detection result of the vehicle direction indicator signal.

[0056] The following will exemplarily describe the configuration and detection method of the fourth detection unit 107 in Embodiment 1 of the present invention.

[0057] FIG. 4 is a schematic diagram of one of the fourth detection units in Embodiment 1 of the present invention. As shown in FIG. 4, the fourth detection unit 107 includes a second determination unit 401, a fifth detection unit 402, and a third determination unit 403.

[0058] When the second determination unit 401 detects that only one of a pair of lamps blinks within one frame period, it determines that the signal of the blinking lamp is a signal for the vehicle to turn right or left. When the fifth detection unit 402 detects that both of the pair of lamps blink within one frame period, it detects the blinking state of the pair of lamps within a plurality of frame periods including the one frame period.

[0059] When the third determination unit 403 detects that only one of the pair of lamps continues to blink within the plurality of frame periods, it determines that the signal of the blinking lamp is a signal for the vehicle to turn right or left. When the third determination unit 403 detects that both of the pair of lamps continue to blink within the plurality of frame periods, it determines that the signal of the blinking lamp is a signal of the hazard lamp.

[0060] In one aspect of an embodiment of the present invention, when there are a plurality of lamp pairs, the fourth detection unit 107 performs detection for each pair.

[0061] In one aspect of an embodiment of the present invention, as shown in FIG. 4, the fourth detection unit 107 may further include a sixth detection unit 404.

[0062] The sixth detection unit 404 detects whether the lamp is blinking based on the luminance change level in the location area of the lamp. The luminance change level in the location area of the lamp is determined based on the above luminance extraction result. In other words, the luminance change level is calculated based on the change in the extracted luminance, and it is detected whether the lamp is blinking based on the luminance change level.

[0063] FIG. 5 is a schematic diagram of one of the detection methods of the fourth detection unit according to the first embodiment of the present invention. As shown in FIG. 5, for a pair of lamps, the method includes the following steps.

[0064] Step 501: Detect the blinking state of a pair of lamps within one frame period.

[0065] Step 502: Determine whether blinking of the lamp is detected within one frame period. If the determination result is "YES", proceed to step 503. If the determination result is "NO", end the process.

[0066] Step 503: Determine whether blinking of only one of the pair of lamps is detected within one frame period. If the determination result is "YES", proceed to step 504. If the determination result is "NO", that is, if it is detected that both of the pair of lamps are blinking, proceed to step 505.

[0067] Step 504: Determine that the signal of the blinking lamp is a signal for a right or left turn of the vehicle.

[0068] Step 505: Detect the blinking state of the pair of lamps within a plurality of frame periods including the one frame period.

[0069] Step 506: Determine whether it is detected that only one of the pair of lamps continues to blink within the plurality of frame periods. If the determination result is "YES", proceed to step 504. If the determination result is "NO", that is, if it is detected that both lamps continue to blink, proceed to step 507.

[0070] Step 507: Determine that the signal of the flashing lamp is the signal of the hazard lamp.

[0071] By performing detection according to different conditions of single-sided flashing and double-sided flashing, accurate detection results of the signal of the direction indicator can be obtained flexibly and efficiently.

[0072] In step 506, when determining whether it is detected that only one lamp out of the pair of lamps continues to flash within the plurality of frame periods, for example, the following method may be used to determine whether the lamp continues to flash. In three frame periods, with one frame period as a unit, the detection window moves continuously or intermittently. For example, the three frame periods include 75 frames numbered from 1 to 75, and a total of 51 flashing detections are performed within the frames numbered 1 to 25, 2 to 26, 3 to 27,..., 51 to 75. If it is detected that only one lamp out of the pair of lamps flashes in N of the flashing detections, when N / 51 is greater than a predetermined threshold, it is determined that the lamp out of the pair of lamps continues to flash.

[0073] For example, when the luminance change of a lamp within one frame period exceeds a predetermined threshold, it is determined that the lamp flashes within this one frame period. The predetermined threshold may be set according to actual requirements. For example, the predetermined threshold is 20%.

[0074] In one aspect of the embodiment of the present invention, the first threshold may be determined according to actual requirements. For example, the first threshold is 80%.

[0075] According to this embodiment, the detection of the lamp is performed based on the vehicle detection result, that is, by using the detection method instead of the conventional clustering method, the accuracy of the lamp detection can be improved. When performing the pairing of the lamps, by combining and considering the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, the accuracy and efficiency of the lamp pairing can be improved. Therefore, the signal of the direction indicator of the vehicle can be recognized in various environments, with a high recognition accuracy and a wide application range.

[0076] <Example 2> The embodiment of the present invention further provides an electronic device. FIG. 6 is a schematic diagram of one of the electronic devices according to Embodiment 2 of the present invention. As shown in FIG. 6, the electronic device 600 includes a recognition device 601 for the signal of the direction indicator of the vehicle. The configuration and function of the recognition device 601 for the signal of the direction indicator of the vehicle are the same as those described in Embodiment 1, and the description thereof is omitted here.

[0077] In this embodiment, the electronic device 600 may be various electronic devices, for example, an in-vehicle terminal, a mobile terminal, or a computer.

[0078] FIG. 7 is a schematic block diagram of the system configuration of the electronic device according to Embodiment 2 of the present invention. As shown in FIG. 7, the electronic device 700 may further include a processor 701 and a memory 702, and the memory 702 is connected to the processor 701. This figure is merely exemplary, and other types of configurations may be used to supplement or replace this configuration so as to realize the electric communication function or other functions.

[0079] As shown in FIG. 7, the electronic device 700 may further include an input unit 703, a display 704, and a power supply 705.

[0080] In one aspect, the function of the recognition device for the turn signal indication of the vehicle in Embodiment 1 may be integrated into the processor 701. Here, the processor 701 sequentially performs vehicle detection for each frame in the input frame sequence, where the frame sequence includes at least one frame period, and steps of performing vehicle tracking based on the vehicle detection results of each frame to obtain a vehicle tracking result, performing lamp detection based on the vehicle detection results of each frame to obtain a lamp detection result, performing detection of the display direction of the vehicle based on the vehicle detection results of each frame to obtain a vehicle display direction detection result, performing lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result to obtain a lamp pairing result, extracting the luminance of the lamp location area based on the lamp pairing result to obtain a luminance extraction result, and performing detection of the turn signal indication of the vehicle based on the luminance extraction result to obtain a vehicle turn signal indication detection result. It may be configured to execute these steps.

[0081] For example, the step of sequentially performing vehicle detection for each frame in the input frame sequence includes the step of sequentially performing vehicle detection for each frame in the input frame sequence using a Feature Pyramid Networks (FPN).

[0082] For example, the step of performing vehicle tracking based on the vehicle detection results of each frame to obtain a vehicle tracking result includes comparing the vehicle detected in the current frame with the vehicles tracked in a predetermined number of frames before the current frame, calculating the overlap ratio of the two compared vehicles, and when the overlap ratio of the vehicle detected in the current frame and the vehicles tracked in a predetermined number of frames before the current frame is greater than a first threshold, setting the vehicle detected in the current frame as the vehicle tracking result of the current frame.

[0083] For example, the step of detecting a lamp based on the vehicle detection result of each frame includes the step of detecting a lamp using the YOLO V3 network based on the vehicle detection result of each frame.

[0084] For example, the step of detecting the display direction of a vehicle based on the vehicle detection result of each frame includes the step of detecting the display direction of the vehicle using a classifier for the vehicle display direction based on the vehicle detection result of each frame, and the classifier for the vehicle display direction is trained by a deep learning method.

[0085] For example, the step of detecting a signal of the vehicle's direction indicator based on the luminance extraction result and obtaining a detection result of the vehicle direction indicator signal includes: when it is detected that only one of a pair of lamps blinks within one frame period, determining that the signal of the blinking lamp is a signal for a right or left turn of the vehicle; when it is detected that both of the pair of lamps blink within one frame period, detecting the blinking state of the pair of lamps within a plurality of frame periods including the one frame period; when it is detected that only one of the pair of lamps continues to blink within the plurality of frame periods, determining that the signal of the blinking lamp is a signal for a right or left turn of the vehicle; and when it is detected that both of the pair of lamps continue to blink within the plurality of frame periods, determining that the signal of the blinking lamp is a signal for a hazard lamp.

[0086] For example, it further includes the step of detecting whether the lamp is blinking based on the luminance change level of the area where the lamp is located, and the luminance change level of the area where the lamp is located is determined based on the luminance extraction result.

[0087] In another aspect, the recognition device for the signal of the direction indicator of the vehicle in Embodiment 1 may be arranged separately from the processor 701. For example, the recognition device for the signal of the direction indicator of the vehicle may be a chip connected to the processor 701, and may be configured to realize the function of the recognition device for the signal of the direction indicator of the vehicle under the control of the processor 701.

[0088] The electronic device 700 in this embodiment does not necessarily include all the components shown in FIG. 7.

[0089] As shown in FIG. 7, the processor 701, also referred to as a controller or an operation control unit, may include a microprocessor or other processing device and / or logic device. The processor 701 receives inputs and controls the operations of the various parts of the electronic device 700.

[0090] The memory 702 may be, for example, one or more of a buffer, a flash memory, a hard disk, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. Further, the processor 701 may execute the program stored in the memory 702 to realize storage or processing of information, etc. Since other members are similar to the prior art, the description thereof is omitted here. Each part of the electronic device 700 may be realized by dedicated hardware, firmware, software, or a combination thereof without departing from the scope of the present invention.

[0091] According to this embodiment, by performing lamp detection based on the vehicle detection result, that is, by using a detection method instead of the conventional clustering method, the accuracy of lamp detection can be improved. When performing lamp pairing, by considering the combination of the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, the accuracy and efficiency of lamp pairing can be improved. Therefore, the signal of the direction indicator of the vehicle can be recognized in various environments, with high recognition accuracy and a wide application range.

[0092] <Embodiment 3> An embodiment of the present invention further provides a method for recognizing a signal of a vehicle direction indicator corresponding to the signal recognition device of the vehicle direction indicator in Embodiment 1. FIG. 8 is a schematic diagram of a method for recognizing a signal of a vehicle direction indicator according to Embodiment 3 of the present invention. As shown in FIG. 8, the method includes the following steps.

[0093] Step 801: Sequentially perform vehicle detection for each frame in the input frame sequence. The frame sequence includes at least one frame period.

[0094] Step 802: Perform vehicle tracking based on the vehicle detection result of each frame, and obtain a vehicle tracking result.

[0095] Step 803: Perform lamp detection based on the vehicle detection result of each frame, and obtain a lamp detection result.

[0096] Step 804: Perform detection of the display direction of the vehicle based on the vehicle detection result of each frame, and obtain a vehicle display direction detection result.

[0097] Step 805: Perform lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, and obtain a lamp pairing result.

[0098] Step 806: Extract the luminance of the lamp location area based on the lamp pairing result, and obtain a luminance extraction result.

[0099] Step 807: Perform detection of the signal of the vehicle direction indicator based on the luminance extraction result, and obtain a detection result of the signal of the vehicle direction indicator.

[0100] In this embodiment, the specific implementation methods of the above steps are the same as those described in Embodiment 1, and the description thereof is omitted here.

[0101] This embodiment is not limited to the execution order of Step 802, Step 803, and Step 804, and they may be executed in parallel or one by one.

[0102] According to this embodiment, by performing lamp detection based on the vehicle detection result, that is, by using a detection method instead of the conventional clustering method, the accuracy of lamp detection can be improved. When performing lamp pairing, by combining and considering the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result, the accuracy and efficiency of lamp pairing can be improved. Therefore, in various environments, the signal of the vehicle's direction indicator can be recognized with high recognition accuracy and a wide application range.

[0103] An embodiment of the present invention further provides a computer-readable program for causing a computer to execute the method for recognizing the signal of the vehicle's direction indicator described in Embodiment 3 in a device or electronic device for recognizing the signal of the vehicle's direction indicator when executing a program.

[0104] An embodiment of the present invention further provides a storage medium for storing a computer-readable program for causing a computer to execute the method for recognizing the signal of the vehicle's direction indicator described in Embodiment 3 in a device or electronic device for recognizing the signal of the vehicle's direction indicator.

[0105] The method for recognizing the signal of the vehicle's direction indicator executed in the device or electronic device for recognizing the signal of the vehicle's direction indicator described with reference to the embodiments of the present invention may be implemented by hardware, a software module executed by a processor, or a combination of both. For example, one or more of the functional block diagrams shown in FIG. 1, or a combination of one or more of the functional block diagrams, may correspond to each software module of the computer program flow, or may correspond to each hardware module. These software modules may each correspond to the steps shown in FIG. 8. These hardware modules may be realized by hardware-implementing these software modules using, for example, a field programmable gate array (FPGA).

[0106] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM (Registered Trademark) memory, a register, a hard disk, a mobile hard disk, a CD-ROM, or any other form of storage medium known to those skilled in the art. The storage medium may be connected to the processor so that the processor can read information from the storage medium or write information to the storage medium, or the storage medium may be a component of the processor. The processor and the storage medium are located in an ASIC. The software module may be stored in the memory of the mobile terminal or may be stored in a memory card inserted into the mobile terminal. For example, when a device (such as a mobile terminal) uses a relatively large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0107] One or more of the functional blocks described in FIG. 1 and / or one or more combinations of the functional blocks may be implemented by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, or any suitable combination thereof for performing the functions described in this application. One or more of the functional blocks described in FIG. 1 and / or one or more combinations of the functional blocks may be implemented, for example, by a combination of computing devices, such as a combination of a DSP and a microprocessor, a combination of multiple microprocessors, one or more microprocessors in combination with DSP communication, or any other configuration.

[0108] The present invention has been described with reference to specific embodiments, but the above description is merely illustrative and does not limit the scope of protection of the present invention. Without departing from the spirit and principle of the present invention, various modifications and changes may be made to the present invention, and these modifications and changes also belong to the scope of the present invention.

[0109] Furthermore, the following supplementary notes are disclosed regarding the embodiments including the above-described examples. (Supplementary Note 1) An apparatus for recognizing a signal of a vehicle direction indicator, a first detection unit that sequentially performs vehicle detection for each frame in the input frame sequence, where the frame sequence includes at least one frame period, the first detection unit; a tracking unit that performs vehicle tracking based on the vehicle detection results of each frame and obtains a vehicle tracking result; a second detection unit that performs lamp detection based on the vehicle detection results of each frame and obtains a lamp detection result; a third detection unit that performs detection of the display direction of the vehicle based on the vehicle detection results of each frame and obtains a vehicle display direction detection result; a matching unit that performs lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result and obtains a lamp pairing result; an extraction unit that extracts luminance for the location area of the lamp based on the lamp pairing result and obtains a luminance extraction result; and a fourth detection unit that performs detection of a signal of the vehicle direction indicator based on the luminance extraction result and obtains a detection result of the vehicle direction indicator signal, the apparatus comprising. (Supplementary Note 2) The apparatus according to Supplementary Note 1, wherein the first detection unit sequentially performs vehicle detection for each frame in the input frame sequence using a Feature Pyramid Network (FPN). (Supplementary Note 3) The tracking unit is A comparison unit that compares the vehicle detected in the current frame with the vehicles tracked in a predetermined number of frames before the current frame and calculates the overlap ratio of the two compared vehicles; A first determination unit that, when the overlap ratio between the vehicle detected in the current frame and the vehicles tracked in a predetermined number of frames before the current frame is greater than a first threshold value, sets the vehicle detected in the current frame as the vehicle tracking result of the current frame, the apparatus according to appended note 1. (Appended note 4) The second detection unit performs detection of a lamp using the YOLO V3 network based on the vehicle detection result of each frame, the apparatus according to appended note 1. (Appended note 5) The third detection unit performs detection of the display direction of a vehicle using a classifier for the vehicle display direction based on the vehicle detection result of each frame, wherein the classifier for the vehicle display direction is trained by a deep learning method, the apparatus according to appended note 1. (Appended note 6) The fourth detection unit a second determination unit that, when it is detected that only one of a pair of lamps blinks within one frame period, determines that the signal of the blinking lamp is a signal for a right or left turn of the vehicle; a fifth detection unit that, when it is detected that both of a pair of lamps blink within one frame period, detects the blinking state of the pair of lamps within a plurality of frame periods including the one frame period; a third determination unit that, when it is detected that only one of the pair of lamps continues to blink within the plurality of frame periods, determines that the signal of the blinking lamp is a signal for a right or left turn of the vehicle, and when it is detected that both of the pair of lamps continue to blink within the plurality of frame periods, determines that the signal of the blinking lamp is a signal for a hazard lamp, the apparatus according to appended note 1. (Appended note 7) The fourth detection unit further includes a sixth detection unit that detects whether the lamp is blinking based on the luminance change level of the area where the lamp is located. The device according to Supplementary Note 6, wherein the luminance change level in the location area of the lamp is determined based on the luminance extraction result. (Supplementary Note 8) An electronic device including the device according to Supplementary Note 1. (Supplementary Note 9) A method for recognizing a signal of a vehicle direction indicator, comprising: sequentially performing vehicle detection for each frame in the input frame sequence, wherein the frame sequence includes at least one frame period; performing vehicle tracking based on the vehicle detection result of each frame to obtain a vehicle tracking result; performing lamp detection based on the vehicle detection result of each frame to obtain a lamp detection result; performing detection of the display direction of the vehicle based on the vehicle detection result of each frame to obtain a vehicle display direction detection result; performing lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle display direction detection result to obtain a lamp pairing result; extracting luminance for the location area of the lamp based on the lamp pairing result to obtain a luminance extraction result; performing detection of a signal of the vehicle direction indicator based on the luminance extraction result to obtain a detection result of the signal of the vehicle direction indicator. (Supplementary Note 10) The step of sequentially performing vehicle detection for each frame in the input frame sequence is: The method according to Supplementary Note 9, comprising sequentially performing vehicle detection for each frame in the input frame sequence using a Feature Pyramid Networks (FPN). (Supplementary Note 11) The step of performing vehicle tracking based on the vehicle detection result of each frame to obtain a vehicle tracking result is: Comparing the vehicle detected in the current frame with the vehicles tracked in a predetermined number of frames preceding the current frame, and calculating the overlap ratio of the two compared vehicles; If the overlap ratio of the vehicle detected in the current frame and the vehicles tracked in a predetermined number of frames preceding the current frame is greater than a first threshold value, using the vehicle detected in the current frame as the vehicle tracking result of the current frame; The method according to Supplementary Note 9, comprising: (Supplementary Note 12) The step of detecting a lamp based on the vehicle detection result of each frame is The method according to Supplementary Note 9, comprising: detecting a lamp using the YOLO V3 network based on the vehicle detection result of each frame. (Supplementary Note 13) The step of detecting the display direction of a vehicle based on the vehicle detection result of each frame is The method according to Supplementary Note 9, comprising: detecting the display direction of a vehicle using a classifier for vehicle display directions based on the vehicle detection result of each frame, The classifier for vehicle display directions is trained by a deep learning method. The method according to Supplementary Note 9. (Supplementary Note 14) The step of detecting a signal of a direction indicator of a vehicle based on the luminance extraction result and obtaining a detection result of the signal of the vehicle direction indicator is When it is detected that only one lamp out of a pair of lamps blinks within one frame period, determining that the signal of the blinking lamp is a signal for a right or left turn of the vehicle; When it is detected that both lamps out of a pair of lamps blink within one frame period, detecting the blinking state of the pair of lamps within a plurality of frame periods including the one frame period; When it is detected that only one of the pair of lamps continues to blink within the plurality of frame periods, it is determined that the signal of the blinking lamp is a signal for a right or left turn of the vehicle. When it is detected that both of the pair of lamps continue to blink within the plurality of frame periods, it is determined that the signal of the blinking lamps is a signal for hazard lamps. The method according to Supplementary Note 9, comprising the steps of (Supplementary Note 15) The step of detecting a signal of a direction indicator of a vehicle based on the luminance extraction result and obtaining a detection result of the vehicle direction indicator signal is Further comprising the step of detecting whether the lamp is blinking based on a luminance change level in a region where the lamp is located, The method according to Supplementary Note 14, wherein the luminance change level in the region where the lamp is located is determined based on the luminance extraction result.

Claims

1. A recognition device for a signal of a vehicle direction indicator, a first detection unit that sequentially performs vehicle detection for each frame in a frame sequence captured by an in-vehicle camera or a road monitoring camera, where the frame sequence includes at least one frame period, and the first detection unit; a tracking unit that performs vehicle tracking based on the vehicle detection results of each frame and obtains a vehicle tracking result, where the vehicle detection results are the detection frames and corresponding position information of each detected vehicle, and the tracking unit; a second detection unit that performs lamp detection within the detection frame of each vehicle based on the vehicle detection results of each frame and obtains a lamp detection result; a third detection unit that performs detection of the shooting direction of the vehicle based on the vehicle detection results of each frame and obtains a vehicle shooting direction detection result; a matching unit that performs lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle shooting direction detection result and obtains a lamp pairing result, where the vehicle tracking result represents the continuously detected result of the vehicle in the frame sequence, the lamp detection result represents the lamp area further detected in the detection frame of the vehicle, and the vehicle shooting direction detection result represents the shooting direction of the vehicle displayed in the vehicle image, and the matching unit; an extraction unit that extracts the luminance of the lamp location area based on the lamp pairing result, performs binarization processing on the image with the extracted luminance based on a predetermined threshold for a daytime scene or a nighttime scene, and obtains a luminance extraction result; a fourth detection unit that performs detection of the signal of the vehicle direction indicator based on the luminance extraction result after the binarization processing and obtains a detection result of the vehicle direction indicator signal. A recognition device for a signal of a vehicle direction indicator, including the above components.

2. The recognition device for a signal of a vehicle direction indicator according to Claim 1, where the first detection unit is a feature pyramid network that takes each frame in the frame sequence as an input and outputs the vehicle detection result.

3. The tracking unit is as follows: a comparison unit that compares the vehicle detected in the current frame with the vehicles tracked in a predetermined number of previous frames of the current frame, and calculates the overlap ratio of the two compared vehicles. When the overlapping rate between the vehicle detected in the current frame and the vehicle tracked in a plurality of predetermined frames before the current frame is greater than a first threshold value, a first determination unit that uses the vehicle detected in the current frame as the vehicle tracking result in the current frame; The vehicle direction indicator signal recognition device according to claim 1, comprising:

4. The vehicle direction indicator signal recognition device according to claim 1, wherein the second detection unit is a YOLO V3 network that takes the vehicle detection result as an input and outputs the lamp detection result.

5. The third detection unit is a classifier of the vehicle shooting direction that takes the vehicle detection result as an input and outputs the shooting direction of the vehicle. The vehicle direction indicator signal recognition device according to claim 1, wherein the classifier of the vehicle shooting direction is trained by a deep learning method.

6. The fourth detection unit A second determination unit that, when it is detected that only one of a pair of lamps on the front side or the rear side of the vehicle blinks within one frame period, determines that the signal of the blinking lamp is a signal for a right or left turn of the vehicle; A fifth detection unit that, when it is detected that both of the pair of lamps blink within one frame period, detects the blinking state of the pair of lamps within a plurality of frame periods including the one frame period; The vehicle direction indicator signal recognition device according to claim 1, further comprising: a third determination unit that, when it is detected that only one of the pair of lamps continues to blink within the plurality of frame periods, determines that the signal of the blinking lamp is a signal for a right or left turn of the vehicle; and when it is detected that both of the pair of lamps continue to blink within the plurality of frame periods, determines that the signal of the blinking lamp is a signal of a hazard lamp.

7. The fourth detection unit Further comprising: a sixth detection unit that detects whether the lamp is blinking based on a luminance change level in a region where the lamp is located. The vehicle direction indicator signal recognition device according to claim 6, wherein the luminance change level in the region where the lamp is located is determined based on the luminance extraction result.

8. An electronic device including the vehicle direction indicator signal recognition device according to claim 1.

9. A method for recognizing a signal of a vehicle direction indicator, comprising: A step of sequentially performing vehicle detection for each frame in a frame sequence captured by an in-vehicle camera or a road monitoring camera, wherein the frame sequence includes at least one frame period, the step; A step of performing vehicle tracking based on the vehicle detection results of each frame and obtaining a vehicle tracking result, wherein the vehicle detection results are the detection frames and corresponding position information of each detected vehicle, the step; A step of performing lamp detection within the detection frame of each vehicle based on the vehicle detection results of each frame and obtaining a lamp detection result; A step of performing detection of the shooting direction of the vehicle based on the vehicle detection results of each frame and obtaining a vehicle shooting direction detection result; A step of performing lamp pairing based on the vehicle tracking result, the lamp detection result, and the vehicle shooting direction detection result and obtaining a lamp pairing result, wherein the vehicle tracking result represents the continuously detected result of the vehicle in the frame sequence, the lamp detection result represents the lamp area further detected in the detection frame of the vehicle, and the vehicle shooting direction detection result represents the shooting direction of the vehicle displayed in the vehicle image, the step; A step of extracting the luminance of the lamp location area based on the lamp pairing result, performing binarization processing on the image from which the luminance is extracted based on a predetermined threshold for a daytime scene or a nighttime scene, and obtaining a luminance extraction result; A step of detecting a signal of the vehicle's direction indicator based on the luminance extraction result after the binarization processing and obtaining a detection result of the vehicle's direction indicator signal, including a method for recognizing a signal of the vehicle's direction indicator.

10. The step of detecting a signal of the vehicle's direction indicator based on the luminance extraction result and obtaining a detection result of the vehicle's direction indicator signal is: When it is detected that only one of a pair of lamps on the front side or the rear side of the vehicle blinks within one frame period, determining that the signal of the blinking lamp is a signal for the vehicle to turn right or left; When it is detected that both of the pair of lamps blink within one frame period, detecting the blinking state of the pair of lamps within a plurality of frame periods including the one frame period; When it is detected that only one of the pair of lamps blinks continuously within the plurality of frame periods, it is determined that the signal of the blinking lamp is a signal for a right or left turn of the vehicle. When it is detected that both of the pair of lamps blink continuously within the plurality of frame periods, it is determined that the signal of the blinking lamps is a signal for hazard lamps. The method for recognizing a signal of a direction indicator of a vehicle according to claim 9, comprising the step of

Images