Method, device and equipment for collecting training data, vehicle and medium

Abstract

The embodiment of the invention relates to a method, device and equipment for collecting training data, a vehicle and a medium. The method includes determining a first degree of fatigue of a driver of the vehicle based on driving data related to driving of the vehicle. The method further includes determining a second degree of fatigue of the driver based on the sequence of facial images of the driver. The method further includes determining whether the first fatigue degree is consistent with the second fatigue degree, and collecting at least one of the target driving data and the target facial image sequence within a predetermined time period as training data when the first fatigue degree is inconsistent with the second fatigue degree. Through the method provided by the embodiment of the invention, more effective training data can be obtained, the cost of obtaining the effective training data can be reduced, and the performance of various fatigue detection systems can be further improved by using the training data.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of data processing, and in particular to methods, devices, equipment, vehicles, and media for collecting training data. Background Art

[0002] With the increase in the number of vehicles, more and more people drive vehicles to reach different destinations. However, some drivers may experience fatigue driving during long-distance driving. Even short-distance driving may cause fatigue driving problems, and the proportion of traffic accidents caused by fatigue driving is also gradually increasing.

[0003] In order to avoid the problem of fatigue driving as much as possible, many technologies for detecting driver fatigue have emerged. It uses various sensors and algorithms to analyze the driver's behavior and physiological state to determine whether they are in a state of fatigue or inattention. Once the system detects signs of driver fatigue or inattention, it can sound an alarm to attract the driver's attention and take necessary measures, such as taking a break or alternating driving. This helps improve road safety and prevent accidents caused by driver fatigue. However, there are still many technical problems that need to be solved in the technology of detecting driver fatigue. Summary of the invention

[0004] Embodiments of the present disclosure provide a method, apparatus, device, vehicle, and medium for collecting training data.

[0005] According to a first aspect of the present disclosure, a method for collecting training data is provided. The method includes determining a first fatigue level of a driver of a vehicle based on driving data related to driving of the vehicle. The method also includes determining a second fatigue level of the driver based on a facial image sequence of the driver. The method also includes determining whether the first fatigue level is consistent with the second fatigue level. The method also includes collecting at least one of target driving data and a target facial image sequence within a predetermined period as training data if it is determined that the first fatigue level and the second fatigue level are inconsistent.

[0006] According to a second aspect of the present disclosure, a device for collecting training data is provided. The device includes a first fatigue level determination unit, configured to determine a first fatigue level of a driver of a vehicle based on driving data related to driving of the vehicle; a second fatigue level determination unit, configured to determine a second fatigue level of the driver based on a facial image sequence of the driver; a consistency determination unit, configured to determine whether the first fatigue level is consistent with the second fatigue level; and a data collection unit, configured to collect at least one of target driving data and a target facial image sequence within a predetermined period as training data if it is determined that the first fatigue level and the second fatigue level are inconsistent.

[0007] According to a third aspect of the present disclosure, an electronic device is provided. The electronic device includes at least one processor; and a memory coupled to the at least one processor and having instructions stored thereon, which, when executed by the at least one processor, causes the device to perform the steps of the method in the first aspect of the present disclosure.

[0008] According to a fourth aspect of the present disclosure, a vehicle is provided. The vehicle includes the electronic device in the third aspect of the present disclosure.

[0009] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored on the computer-readable storage medium, wherein the computer-executable instructions are executed by a processor to implement the steps of the method in the first aspect of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0011] Figure 1 A schematic diagram illustrating an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;

[0012] Figure 2 A flowchart of a method for collecting training data according to an embodiment of the present disclosure is illustrated;

[0013] Figure 3 A schematic diagram illustrating a framework of a driver monitoring system according to an embodiment of the present disclosure is illustrated;

[0014] Figure 4 A schematic diagram illustrating a framework of a driver fatigue detection system according to an embodiment of the present disclosure is illustrated;

[0015] Figure 5 A schematic diagram illustrating an example process of acquiring training data according to an embodiment of the present disclosure is illustrated;

[0016] Figure 6 A schematic diagram illustrating another example process of acquiring training data according to an embodiment of the present disclosure is illustrated;

[0017] Figure 7 A schematic diagram illustrating another example process of acquiring training data according to an embodiment of the present disclosure is illustrated;

[0018] Figure 8 A schematic diagram of an apparatus for collecting training data according to an embodiment of the present disclosure is illustrated;

[0019] Fig. 9 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is illustrated.

[0020] In the various drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0021] The embodiments of the present disclosure described below with reference to the accompanying drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure. In addition, before using the technical solutions disclosed in the embodiments of the present disclosure, the type, scope of use, and usage scenarios of the personal information involved in the present disclosure shall be informed to the user in an appropriate manner in accordance with relevant laws and regulations and the user's authorization shall be obtained.

[0022] As mentioned above, in order to avoid driver fatigue driving, many technologies for detecting driver fatigue have emerged. One detection technology is to use a driver monitoring system (DMS) to monitor and analyze vehicle driver behavior. Usually, the analysis data of the driver monitoring system is collected through a camera. Another detection technology is to use a driver fatigue detection system (DDD) to monitor and analyze vehicle driver behavior. This technology mainly determines the driver's fatigue level by analyzing the steering wheel behavior.

[0023] Typically, some fatigue detection systems can be implemented by machine learning models. However, the collection of driver fatigue data used to train machine learning models is very expensive because it takes hours of driving to trigger. In addition, not all driver fatigue data obtained is valid data. Because the period of fatigue driving is relatively short compared to the period of normal driving, there is too much data collected from the driver's normal driving, resulting in a waste of data storage resources, transmission resources, and computing resources.

[0024] At least to solve the above and other potential problems, an embodiment of the present disclosure provides a method for collecting training data. In this method, an electronic device obtains driving data and a facial image sequence, and then uses two systems / models to process the obtained driving data and facial image sequence respectively to calculate two fatigue levels. Next, the computing device compares the two fatigue levels to determine whether the two fatigue levels are consistent. If they are inconsistent, it indicates that there is a problem with the calculation result of one of the two systems / models. Therefore, it is necessary to collect at least one of the target driving data and the target facial image sequence within a predetermined period of time to use it as training data for adjusting the erroneous system / model of the two systems / models. Through this method, effective training data for training machine models can be determined through the inconsistency of the two fatigue levels, reducing the cost of obtaining effective training data, and the training data can be used to improve the performance of various fatigue detection systems.

[0025] The embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. Figure 1 An example environment is shown in which the apparatus and / or method of embodiments of the present disclosure may be implemented.

[0026] like Figure 1 As shown, the example environment 100 includes a vehicle 102. The vehicle 102 is driven by a driver and travels on a road. The vehicle 102 includes an electronic device 104. The electronic device 104 is used to control the collection of training data. In one example, the electronic device 104 can be a domain controller in the vehicle. In another example, the electronic device 104 can be a controller separated from the domain controller in the vehicle. In some embodiments, the electronic device 104 can be implemented by any suitable computing device, including but not limited to a personal computer, a handheld or laptop device, a mobile device, a multi-processor system, a consumer electronic product, a small computer including a distributed computing environment of any one of the above systems or devices, etc.

[0027] exist Figure 1In the embodiment, driving data 106 related to the driving of the vehicle is transmitted to the electronic device 104. In one example, the driving data 106 related to the driving of the vehicle includes a steering angle signal. For example, the steering angle signal can be obtained from a steering angle sensor and an electric power steering system (Electric Power Steering System) arranged in the vehicle. In another example, the driving data 106 related to the driving of the vehicle includes the speed of the vehicle. The speed of the vehicle can be obtained by a speed sensor, a navigation system or a vehicle speed radar. In yet another example, the driving data 106 related to the driving of the vehicle includes turn indication data. The turn indication data can be obtained from a sensor that senses the steering action. The above examples are only used to describe the present disclosure, not specific data of the present disclosure. The driving data related to the driving of the vehicle may include two or more of the above data, or may also include any other suitable data.

[0028] The electronic device 104 can determine the fatigue level 110 of the vehicle driver using the obtained driving data 106, also referred to as the first fatigue level. In one example, the fatigue level 110 can be indicated by a fatigue level. Generally, different fatigue levels can be represented by multiple fatigue levels. For example, it can be divided into normal, mild fatigue and severe fatigue. Alternatively or additionally, fatigue can be divided into any suitable number of levels and the characteristics corresponding to each level can be determined. In another example, the fatigue level 110 can be indicated by the change trend of the fatigue level. For example, multiple fatigue levels of a threshold period are obtained, and then the change trend of the fatigue level is determined from the multiple fatigue levels. In some embodiments, the electronic device 104 uses a driver fatigue detection system to process the driving data 106 to obtain the fatigue level 110. In one example, the driver fatigue detection system is composed of a machine learning model. In another example, the driver fatigue detection system calculates the data and determines the fatigue level according to certain rules. The above examples are generated only for describing the present disclosure, not for specific limitation of the present disclosure.

[0029] In addition to receiving driving data 106 related to driving of the vehicle, the electronic device 104 can further receive a driver's facial image sequence 108. In some embodiments, the driver's facial image sequence 108 is a series of images taken by a camera. In some embodiments, the driver's facial image sequence 108 is a video taken by a camera. The camera can be arranged on a steering column, an A-pillar, an interior rearview mirror, or a center console.

[0030] The electronic device 104 uses the driver's facial image sequence 108 to determine the fatigue level 112, also referred to as the second fatigue level. Similar to the fatigue level 110, the fatigue level 112 can also be indicated by a fatigue level or a trend of changes in the fatigue level. In one example, the grading of fatigue levels for the second fatigue level is the same as the grading of fatigue levels for the first fatigue level. Therefore, when the first fatigue level and the second fatigue level are the same, they are indicated by the same fatigue level. In another example, the standard for grading fatigue levels for the second fatigue level is different from the standard for grading fatigue levels for the first fatigue level. In order to facilitate the establishment of the correspondence of levels under different standard classification systems, a mapping relationship between different fatigue levels under two classification standards can be set to establish the correspondence of levels under the two fatigue classification standards.

[0031] After obtaining the fatigue level 110 and the fatigue level 112, the electronic device 104 will compare the consistency of the two fatigue levels. If they are consistent, there is no need to collect data at this time. If the fatigue level 110 and the fatigue level 112 are inconsistent, it indicates that there is a problem in one of the two systems or processes in calculating the fatigue level 110 and the fatigue level 112, and a calculation error has occurred. In order to improve the accuracy of these two systems or processes, it is necessary to collect at least one of the target driving data and the target facial image sequence within a predetermined period of time as training data 114. For example, the predetermined period of time is ten minutes from the moment the inconsistency is determined, so data within ten minutes is collected. The above examples are only used to describe the present disclosure, and are not specific limitations on the present disclosure. Alternatively, those skilled in the art can set the predetermined period of time to any suitable duration.

[0032] After the training data 114 is obtained, it can be sent to the model training system 116 to train the driver monitoring system or the driver fatigue detection system. In addition, the collected training data is negative sample data for training the driver monitoring system or the driver fatigue detection system. By further training the above systems with the training data, the accuracy of the predictions of the above two systems can be improved.

[0033] Alternatively or additionally, the process of collecting driving data and facial image sequences to determine consistency and collecting data is implemented through shadow mode, in which the computing device only collects data from the user's actual driving process and performs calculations, and the calculated fatigue driving results do not affect and interfere with the actual user operations.

[0034] Through this method, effective training data for training machine models can be determined through the inconsistency of two fatigue levels, reducing the collection cost of obtaining effective training data, as well as the computing resources and transmission resources for obtaining and transmitting training data. The training data can also be used to improve the performance of the driver monitoring system and the driver fatigue detection system.

[0035] Combined with the above Figure 1 A block diagram of an example system 100 is depicted in which embodiments of the present disclosure can be implemented. Figure 2 A flowchart of a method 200 for collecting training data according to an embodiment of the present disclosure is described. The method 200 may be Figure 1 The process is executed at the electronic device 104 and any suitable computing device.

[0036] At box 202, based on driving data related to driving of the vehicle, a first fatigue level of the driver of the vehicle is determined. For example, the electronic device 104 determines a first fatigue level 110 for the driver of the vehicle 102 according to the obtained driving data 106, which is used to reflect the state of the driver.

[0037] In some embodiments, the first fatigue level is determined by a first model in a fatigue detection system. In one example, the model is a machine learning model, which is trained by sample driving data and sample fatigue levels. The first fatigue level can be obtained by inputting the obtained steering angle signal, the speed of the vehicle and the steering indication data into the machine learning model. Alternatively or additionally, the first model can be used as a driver fatigue detection system. In some embodiments, there is a mapping relationship between driving data and the driver's fatigue level in the driver fatigue detection system of the computing device. The first fatigue level is determined based on the mapping relationship between driving data and the driver's fatigue level. For example, after obtaining the driving data, the driving data is used to check the corresponding fatigue level in the mapping relationship. In some embodiments, the first fatigue level is determined by the fatigue detection system calculating the driving data and determining the first fatigue level according to certain rules, such as calculating the change in the steering angle and the frequency of the change according to the steering angle signal, and comparing it with a predetermined standard threshold to determine the first fatigue level. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0038] In some embodiments, when determining the first fatigue level of the driver of the vehicle, the electronic device directly calculates a fatigue level to indicate the first fatigue level based on the currently acquired driving data using the above method. In some embodiments, when determining the first fatigue level of the driver of the vehicle, the electronic device 104 can use the driving data within the threshold period, such as data within 15 minutes, to calculate the first number of fatigue levels. Then the electronic device uses the first number of fatigue levels to determine the first fatigue level indicating the first fatigue level. In one example, the electronic device averages the first number of fatigue levels acquired to determine the first fatigue level. In another example, the electronic device determines the fatigue level with the largest number of fatigue levels in the first number, and then uses the fatigue level as the first fatigue level. In another example, the electronic device determines the maximum fatigue level from the first number of fatigue levels as the first fatigue level. In another example, the electronic device determines the minimum fatigue level from the first number of fatigue levels as the first fatigue level. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0039] In some embodiments, when determining a first fatigue level indicating a first degree of fatigue, a first change trend for the fatigue level is calculated based on a first number of fatigue levels within a threshold period. For example, based on the level value of the fatigue level, multiple points for the first number of fatigue levels are determined, and then a curve is fitted to the multiple points to obtain a corresponding curve to represent the change trend of the fatigue level. The first change trend can then be used to indicate the first fatigue level.

[0040] At box 204, based on the facial image sequence of the driver, a second fatigue level of the driver is determined. For example, the electronic device 104 determines the fatigue level 112 of the driver of the vehicle 102 based on the acquired facial image sequence 108 to reflect the state of the driver. In one example, the facial image sequence 108 is a facial video shot for the driver. In another example, the facial image sequence 108 is an image sequence composed of images shot for the driver at predetermined time intervals, such as every 5 seconds, or a facial image frame sequence extracted from the shot video at predetermined time intervals.

[0041] In some embodiments, the second fatigue level is determined by a second model in the fatigue detection system. For example, the fatigue detection system includes a driver monitoring system as the second model. In one example, the model is a neural network model. In some embodiments, there is a mapping relationship between facial image sequence data and the driver's fatigue level, and the second fatigue level is determined based on the mapping relationship between the facial image sequence data and the driver's fatigue level. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0042] In some embodiments, when determining the second fatigue level of the driver of the vehicle, the electronic device directly generates a fatigue level to indicate the second fatigue level based on the currently acquired facial image sequence. In some embodiments, when determining the second fatigue level of the driver of the vehicle, the electronic device 104 can use the facial image sequence within the threshold period to calculate the second number of fatigue levels, and then determine the second fatigue level indicating the second fatigue level based on the second number of fatigue levels. In one example, the electronic device averages the acquired second number of fatigue levels to determine the second fatigue level. In another example, the electronic device determines the fatigue level with the largest number of fatigue levels among the second number of fatigue levels, and then uses the fatigue level as the second fatigue level. In another example, the electronic device determines the maximum fatigue level from the second number of fatigue levels as the second fatigue level. In another example, the electronic device determines the minimum fatigue level from the second number of fatigue levels as the second fatigue level. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure.

[0043] In some embodiments, when determining a second fatigue level indicating a second degree of fatigue, a second change trend for the fatigue level is calculated based on a second number of fatigue levels within a threshold period. For example, based on the level value of the fatigue level, multiple points for the second number of fatigue levels are determined, and then a curve is fitted to the multiple points to obtain a corresponding curve to represent the change trend of the fatigue level. The second change trend can then be used to indicate the second fatigue level.

[0044] At block 206, it is determined whether the first fatigue level is consistent with the second fatigue level. After obtaining the fatigue level 110 and the fatigue level 112, the electronic device 104 determines whether the first fatigue level is consistent with the second fatigue level. Which data is collected is determined by determining whether the first fatigue level is consistent with the second fatigue level.

[0045] In some embodiments, the first fatigue degree is indicated by a first fatigue level, the second fatigue degree is indicated by a second fatigue level, and the first fatigue level and the second fatigue level are from the same set of fatigue levels. At this time, when determining whether the first fatigue degree is consistent with the second fatigue degree, it can be directly determined whether the first fatigue level is the same as the second fatigue level. If it is determined that the first fatigue level is the same as the second fatigue level, it is determined that the first fatigue degree and the second fatigue degree are consistent. If it is determined that the first fatigue level is different from the second fatigue level, it is determined that the first fatigue degree and the second fatigue degree are inconsistent.

[0046] In some embodiments, the first fatigue level comes from a first group of fatigue levels, the second fatigue level comes from a second group of fatigue levels, and the first group of fatigue levels is graded differently from the second group of fatigue levels, but the fatigue levels in the two groups of fatigue levels have a predefined mapping relationship, respectively. For example, it is pre-established which fatigue levels in the first group of fatigue levels correspond to which fatigue levels in the second group of fatigue levels to indicate similar or identical fatigue levels. At this time, when determining whether the first fatigue level is consistent with the second fatigue level, the electronic device first determines whether the first fatigue level has a predefined mapping relationship with the second fatigue level. If it is determined that the first fatigue level has a predefined mapping relationship with the second fatigue level, it is determined that the first fatigue level is consistent with the second fatigue level. If it is determined that the first fatigue level has no predefined mapping relationship with the second fatigue level, it is determined that the first fatigue level is inconsistent with the second fatigue level. Table 1 below gives a mapping relationship between two groups of fatigue levels.

[0047] Table 1: Mapping relationship between two groups of fatigue levels

[0048]

[0049] As shown in Table 1, the first group of fatigue levels includes three fatigue levels: fatigue level A, fatigue level B, and fatigue level C. For example, fatigue level A indicates no fatigue, fatigue level B indicates moderate fatigue, and fatigue level C indicates very tired. The second group of fatigue levels includes five fatigue levels: fatigue level 1, fatigue level 2, fatigue level 3, fatigue level 4, and fatigue level 5. For example, fatigue level 1 indicates no fatigue, and the following four fatigue levels represent four levels of fatigue degree. As shown in Table 1, fatigue level A in the first group of fatigue levels has a mapping relationship with fatigue level 1 in the second group of fatigue levels; fatigue level B in the first group of fatigue levels has a mapping relationship with fatigue level 2 and fatigue level 3 in the second group of fatigue levels; fatigue level C in the first group of fatigue levels has a mapping relationship with fatigue level 4 and fatigue level 5 in the second group of fatigue levels. The fatigue degrees represented by the fatigue levels having a mapping relationship are consistent.

[0050] In some embodiments, the first fatigue degree is indicated by a first change trend and the second fatigue degree is indicated by a second change trend. When determining whether the first fatigue degree is consistent with the second fatigue degree, the electronic device determines whether the first change trend and the second change trend are consistent. If it is determined that the first change trend is inconsistent with the second change trend, it is determined that the first fatigue degree and the second fatigue degree are inconsistent. If it is determined that the first change trend is consistent with the second change trend, it is determined that the first fatigue degree and the second fatigue degree are consistent.

[0051] In some embodiments, if it is determined that the first fatigue level and the second fatigue level are consistent, indicating that the two systems for calculating the first fatigue level and the second fatigue level can correctly calculate the results, then no data is collected. Alternatively or additionally, warning information may be provided to the driver, such as text, sound or image prompt information.

[0052] In some embodiments, if it is determined that the first fatigue level is consistent with the second fatigue level, it can be further determined whether to collect data or provide warning information by determining whether the change trend of the fatigue level is consistent. In this process, the first change trend of the first number of fatigue levels and the second change trend of the second number of fatigue levels within the threshold period are first determined, and then it is determined whether the first change trend and the second change trend are consistent. If it is determined that the first change trend is consistent with the second change trend, data is not collected, but warning information can be provided. If it is determined that the first change trend is inconsistent with the second change trend, data can be collected.

[0053] At box 208, if it is determined that the first fatigue level and the second fatigue level are inconsistent, at least one of the target driving data and the target facial image sequence within a predetermined period is collected as training data. After determining that the first fatigue level and the second fatigue level are inconsistent, it indicates that one of the processes of calculating the two fatigue levels is wrong. The data at this time is valid for subsequent adjustment of the model. Therefore, the data within the predetermined period before the inconsistent moment is collected for subsequent training and adjustment of the model.

[0054] In some embodiments, after the training data is collected, it can be determined which of the first fatigue level and the second fatigue level is wrong according to the actual situation. If it is determined that the first fatigue level is wrong, the collected training data is used to train the first model. If it is determined that the second fatigue level is wrong, the collected training data is used to train the second model. In this way, the obtained training data can be used to train the corresponding model, which speeds up the adjustment of the model and improves the accuracy of the model.

[0055] In some embodiments, when collecting at least one of the target driving data and the target facial image sequence within a predetermined period as training data, since the first and second fatigue levels are obtained based on sensor data, the calculation result is relatively accurate. Therefore, if it is determined that the first fatigue level and the second fatigue level are inconsistent, it can be determined that the second fatigue level is wrong, and then the target facial image sequence within the predetermined period is collected as training data. The above examples are only used to describe the present disclosure, but are not specifically limited to the present disclosure.

[0056] In some embodiments, warning information can also be provided to the user in certain cases when the first fatigue level and the second fatigue level are inconsistent. Therefore, when it is determined that the first fatigue level and the second fatigue level are inconsistent, the first fatigue level and the second fatigue level can be weighted. For example, since the first fatigue level is calculated using sensor data or obtained using rule reasoning, its result has a higher accuracy, while the second fatigue level is calculated based on image recognition. Due to the greater environmental influence, such as light, the calculation result may be relatively low in accuracy. Therefore, a larger weight can be assigned to the first fatigue level, and a smaller weight can be assigned to the second fatigue level. Then, the electronic device can determine the third fatigue level based on the weighted first fatigue level and the weighted second fatigue level. The electronic device also needs to compare the third fatigue level with the threshold fatigue level, and if it is determined that the third fatigue level is greater than the threshold fatigue level, provide warning information to the driver. If the third fatigue level is not greater than the threshold fatigue level, the driver is not reminded.

[0057] Through this method, effective training data for the training model can be determined through the inconsistency of two fatigue levels, reducing the collection cost of obtaining effective training data, as well as the computing resources and transmission resources for obtaining and transmitting training data. The training data can also be used to improve the performance of the driver monitoring system and the driver fatigue detection system.

[0058] Combined with the above Figure 2 A flowchart of a method 200 for collecting training data according to an embodiment of the present disclosure is described. Figure 3 A schematic diagram depicting a framework 300 of a driver monitoring system according to an embodiment of the present disclosure is shown.

[0059] like Figure 3 As shown, a facial image sequence 302 captured by a camera in a vehicle is input into a driver monitoring system 304. An image analysis module 306 of the driver monitoring system 304 performs image analysis on the facial image sequence 302 to extract facial features 308 for the driver. For example, various suitable face recognition algorithms are used to determine the eye features, facial expression features, etc. of a person. Then, the facial features 308 are input into a first fatigue detection module 310 to determine the degree of fatigue of the driver. Both the image analysis module 306 and the first fatigue detection module 310 can be neural network models trained by sample data. During the training process, the driver monitoring system 304 is trained as a whole. At this time, the sample facial image sequence can be input into the driver monitoring system 304, and then the parameters are adjusted according to the sample fatigue degree corresponding to the sample facial image sequence to obtain a trained driver monitoring system.

[0060] Combined with the above Figure 3A schematic diagram of a framework 300 of a driver monitoring system according to an embodiment of the present disclosure is described below. Figure 4 A schematic diagram depicting a framework 400 of a driver fatigue detection system according to an embodiment of the present disclosure.

[0061] like Figure 4 As shown, driving data 402 is input into a driver fatigue detection system 404, and the driving data 402 includes a steering angle signal and a vehicle speed and steering indication data. A data analysis module 406 in the driver fatigue detection system 404 determines the change information of the steering angle in the driving data, the change of the vehicle speed, the change of the acceleration, etc. Then, the analyzed data is sent to a second fatigue detection module 408 to determine the degree of driving fatigue. Figure 4 The data analysis module 406 and the second fatigue detection module 408 shown can be implemented by a neural network model, or by a predetermined calculation formula and in combination with a predetermined determination rule. In the method using the calculation formula, the sliding window method is used to calculate the sliding mean of the steering angle in each sliding window, and then the inflection point of the sliding mean of the steering angle is calculated, and the fatigue degree is further determined in combination with the speed of the vehicle.

[0062] Combined with the above Figure 4 A schematic diagram of a framework 400 of a driver fatigue detection system according to an embodiment of the present disclosure is described below. Figure 5 A schematic diagram depicting an example process 500 for acquiring training data according to an embodiment of the present disclosure.

[0063] like Figure 5 As shown, driving data 502 includes a steering angle signal and the speed and steering indication data of the vehicle. The driving data 502 is then input into a driver fatigue detection system 504 to determine a first fatigue level 506 of the driver of the vehicle. Similarly, a facial image sequence 510 of the driver is input into a driver monitoring system 512 to obtain a second fatigue level 514. Then, at box 516, the first fatigue level 506 and the second fatigue level 514 are compared to see if they are consistent. If they are consistent, return to box 508 to continue with the following detection. If they are inconsistent, the driving data and the target facial image sequence or facial video are captured at box 518. When capturing the driving data and the target facial image sequence or facial video, the data in a time period with a predetermined duration on the time axis is captured, and then the following detection is continued at box 508.

[0064] Combined with the above Figure 5 A schematic diagram of an example process 500 for obtaining training data according to an embodiment of the present disclosure is described below. Figure 6 A schematic diagram depicting another example process 600 for acquiring training data according to an embodiment of the present disclosure.

[0065] like Figure 6 As shown, Figure 5 Similarly, the driving data 602 is input into the driver fatigue detection system 604 to determine the first fatigue level 606 of the driver of the vehicle. Similarly, the facial image sequence 610 of the driver is input into the driver monitoring system 612 to obtain the second fatigue level 614. Then, at box 616, the first fatigue level 606 and the second fatigue level 614 are compared to see if they are consistent. If not, the driving data and the target facial image sequence or facial video are captured at box 618. When capturing the driving data and the target facial image sequence or facial video, the data in the time period with a predetermined duration on the time axis is captured, and then the subsequent detection is continued at box 608. If consistent, the electronic device also needs to further determine the first change trend for the fatigue level according to the first number of fatigue levels obtained from the driving data within the threshold period, and determine the second change trend for the fatigue level according to the second number of fatigue levels obtained from the facial image sequence within the threshold period. Then, at box 620, the consistency of the first change trend and the second change trend is compared, for example, using the similarity of the curves to determine whether the change trends are consistent. If it is determined that the first change trend is consistent with the second change trend, return to box 608 to continue the subsequent detection. If it is determined that the first change trend is inconsistent with the second change trend, data can be collected as training data. For example, the first change trend can be represented by a first curve, and the second change trend can be represented by a second curve. Then, a method such as Frechet or Hausdorff can be used to determine the similarity of the curves. If the similarity of the curves exceeds a threshold similarity, it is determined that the two curves match, otherwise it is determined that the two curves do not match.

[0066] Combined with the above Figure 6 A schematic diagram of another example process 600 for obtaining training data according to an embodiment of the present disclosure is described below. Figure 7 A schematic diagram illustrating yet another example process 700 for acquiring training data according to an embodiment of the present disclosure.

[0067] like Figure 7As shown, driving data 702 within a threshold period is input into a driver fatigue detection system 704 to determine a plurality of fatigue levels 706 of the driver of the vehicle. A first change trend 714 of the plurality of fatigue levels 706 is then calculated. For example, a curve fitting method is used to determine the change trend 714 of the plurality of fatigue levels. Similarly, a facial image sequence 710 of the driver within the threshold period is input into a driver monitoring system 712 to obtain a plurality of fatigue levels 716. A second change trend 720 of the plurality of fatigue levels 716 is then calculated. Then, at box 722, the first change trend 714 and the second change trend 720 are compared to see if they are consistent. If they are consistent, return to box 708 to continue with the following detection. If they are inconsistent, the driving data and the target facial image sequence or facial video are captured at box 718. When capturing the driving data and the target facial image sequence or facial video, data in a time period with a predetermined duration on the time axis is captured, and then the following detection is continued at box 708.

[0068] Figure 8 Further illustrating a schematic diagram of an apparatus for collecting training data according to an embodiment of the present disclosure, the apparatus 800 may be applied to an electronic device 104, which may include multiple modules for performing the following steps: Figure 2 The corresponding steps in the process 200 discussed in Figure 8 As shown, the device 800 includes: a first fatigue level determination unit 802, configured to determine a first fatigue level of a driver of a vehicle based on driving data related to the driving of the vehicle; a second fatigue level determination unit 804, configured to determine a second fatigue level of the driver based on a facial image sequence of the driver; a consistency determination unit 806, configured to determine whether the first fatigue level is consistent with the second fatigue level; and a data collection unit 808, configured to collect at least one of the target driving data and the target facial image sequence within a predetermined period as training data if it is determined that the first fatigue level and the second fatigue level are inconsistent.

[0069] In some embodiments, the first fatigue level is determined by a first model in the fatigue detection system, and the second fatigue level is determined by a second model in the fatigue detection system.

[0070] In some embodiments, the device 800 also includes: a first training unit configured to use the collected training data to train the first model if the first fatigue level is determined to be erroneous; and a first training unit configured to use the collected training data to train the second model if the second fatigue level is determined to be erroneous.

[0071] In some embodiments, the data collection unit 808 includes: a fatigue level error determination unit, configured to determine that the second fatigue level is wrong if the first fatigue level and the second fatigue level are inconsistent; and a target facial image sequence collection unit, configured to collect a target facial image sequence within a predetermined time period as training data.

[0072] In some embodiments, the first fatigue degree determination unit 802 includes: a first number of fatigue levels determination unit, configured to determine a first number of fatigue levels based on driving data within a threshold period; and a first fatigue level determination unit, configured to determine a first fatigue level indicating a first fatigue degree based on the first number of fatigue levels.

[0073] In some embodiments, the second fatigue degree determination unit 804 includes: a second number of fatigue levels determination unit, configured to determine a second number of fatigue levels based on a facial image sequence within a threshold time period; and a second fatigue level determination unit, configured to determine a second fatigue level indicating a second fatigue degree based on the second number of fatigue levels.

[0074] In some embodiments, the first fatigue level and the second fatigue level are from the same group of fatigue levels, and the consistency determination unit 806 includes: a same-level determination unit, configured to determine whether the first fatigue level is the same as the second fatigue level; a first determination unit, configured to determine that the first fatigue degree and the second fatigue degree are consistent if it is determined that the first fatigue level is the same as the second fatigue level; and a first inconsistency determination unit, configured to determine that the first fatigue degree and the second fatigue degree are inconsistent if it is determined that the first fatigue level is different from the second fatigue level.

[0075] In some embodiments, the first fatigue level comes from a first group of fatigue levels, the second fatigue level comes from a second group of fatigue levels, and the fatigue levels in the first group of fatigue levels have a predefined mapping relationship with the fatigue levels in the second group of fatigue levels; wherein the consistency determination unit 806 includes: a mapping relationship determination unit, configured to determine whether the first fatigue level and the second fatigue level have a predefined mapping relationship; a second determination unit, configured to determine that the first fatigue degree and the second fatigue degree are consistent if it is determined that the first fatigue level and the second fatigue level have a predefined mapping relationship; and a second inconsistency determination unit, configured to determine that the first fatigue degree and the second fatigue degree are inconsistent if it is determined that the first fatigue level and the second fatigue level do not have a predefined mapping relationship.

[0076] In some embodiments, the first fatigue level determination unit includes: a first change trend determination unit, configured to determine a first change trend for the fatigue level based on a first number of fatigue levels; and the first fatigue level determination unit includes: a second change trend determination unit, configured to determine a second change trend for the fatigue level based on a second number of fatigue levels.

[0077] In some embodiments, the consistency determination unit 806 includes: a trend consistency determination unit, configured to determine whether the first change trend and the second change trend are consistent; and a third inconsistency determination unit, configured to determine that the first fatigue level and the second fatigue level are inconsistent if it is determined that the first change trend is inconsistent with the second change trend.

[0078] In some embodiments, the device 800 also includes: a weighting unit, configured to weight the first fatigue level and the second fatigue level if it is determined that the first fatigue level and the second fatigue level are inconsistent; a third fatigue level determination unit, configured to determine the third fatigue level based on the weighted first fatigue level and the weighted second fatigue level; and a warning information providing unit, configured to provide warning information to the driver if it is determined that the third fatigue level is greater than a threshold fatigue level.

[0079] Fig. 9 A schematic block diagram of an example device 900 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The electronic device 104 in the embodiment can be implemented by using the device 900. As shown in the figure, the device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or computer program instructions loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0080] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a memory 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0081] The various processes and processing described above, such as method 200 and processes 500, 600, and 700, may be performed by processing unit 901. For example, in some embodiments, method 200 and processes 500, 600, and 700 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of method 200 and processes 500, 600, and 700 described above may be performed.

[0082] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0083] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0084] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0085] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0086] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0087] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0088] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0089] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0090] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for collecting training data, comprising: determining a first fatigue level of a driver of the vehicle based on driving data related to driving of the vehicle; determining a second fatigue level of the driver based on the facial image sequence of the driver; determining whether the first fatigue level is consistent with the second fatigue level; as well as If it is determined that the first fatigue level and the second fatigue level are inconsistent, at least one of the target driving data and the target facial image sequence within a predetermined period of time is collected as training data. 2 . The method of claim 1 , wherein the first fatigue level is determined by a first model in a fatigue detection system, and the second fatigue level is determined by a second model in the fatigue detection system.

3. The method according to claim 2, further comprising: If it is determined that the first fatigue level is wrong, further training the first model using the collected target driving data; as well as If it is determined that the second fatigue level is erroneous, the second model is further trained using the collected target facial image sequence.

4. The method according to claim 2, wherein collecting at least one of target driving data and target facial image sequence within a predetermined period of time as training data comprises: If it is determined that the first fatigue level and the second fatigue level are inconsistent, determining that the second fatigue level is wrong; as well as The target facial image sequence within the predetermined period is collected as the training data.

5. The method of claim 1 , wherein determining a first fatigue level of a driver of the vehicle comprises: determining a first number of fatigue levels based on driving data over a threshold period; as well as Based on the first number of fatigue levels, a first fatigue level indicative of the first degree of fatigue is determined.

6. The method of claim 5, wherein determining the second fatigue level of the driver comprises: a second number of fatigue levels determined based on the sequence of facial images within the threshold period; as well as Based on the second number of fatigue levels, a second fatigue level indicative of the second degree of fatigue is determined.

7. The method of claim 6, wherein the first fatigue level and the second fatigue level are from the same set of fatigue levels, and determining whether the first fatigue level is consistent with the second fatigue level comprises: determining whether the first fatigue level is the same as the second fatigue level; If it is determined that the first fatigue level is the same as the second fatigue level, determining that the first fatigue degree is consistent with the second fatigue degree; as well as If it is determined that the first fatigue level is different from the second fatigue level, it is determined that the first fatigue degree and the second fatigue degree are inconsistent.

8. The method according to claim 6, wherein the first fatigue level is from a first set of fatigue levels, the second fatigue level is from a second set of fatigue levels, and the fatigue levels in the first set of fatigue levels have a predefined mapping relationship with the fatigue levels in the second set of fatigue levels; Determining whether the first fatigue level is consistent with the second fatigue level includes: determining whether the first fatigue level and the second fatigue level have a predefined mapping relationship; If it is determined that the first fatigue level and the second fatigue level have the predefined mapping relationship, determining that the first fatigue degree and the second fatigue degree are consistent; and If it is determined that the first fatigue level and the second fatigue level do not have the predefined mapping relationship, it is determined that the first fatigue degree and the second fatigue degree are inconsistent.

9. The method according to claim 6, Wherein determining a first fatigue level indicating the first fatigue degree comprises: determining a first change trend for fatigue levels based on the first number of fatigue levels; and Determining a second fatigue level indicating the second fatigue degree includes: determining a second change trend for the fatigue level based on the second number of fatigue levels.

10. The method of claim 9, wherein determining whether the first fatigue level is consistent with the second fatigue level comprises: determining whether the first change trend and the second change trend are consistent; as well as If it is determined that the first change trend is inconsistent with the second change trend, it is determined that the first fatigue level is inconsistent with the second fatigue level.

11. The method according to claim 1, further comprising: If it is determined that the first fatigue level and the second fatigue level are inconsistent, weighting the first fatigue level and the second fatigue level; determining a third fatigue level based on the weighted first fatigue level and the weighted second fatigue level; as well as If it is determined that the third fatigue level is greater than a threshold fatigue level, a warning message is provided to the driver.

12. An apparatus for collecting training data, comprising: a first fatigue level determination unit configured to determine a first fatigue level of a driver of the vehicle based on driving data related to driving of the vehicle; a second fatigue level determination unit configured to determine a second fatigue level of the driver based on a facial image sequence of the driver; a consistency determination unit, configured to determine whether the first fatigue level is consistent with the second fatigue level; as well as The data collection unit is configured to collect at least one of the target driving data and the target facial image sequence within a predetermined period as training data if it is determined that the first fatigue level and the second fatigue level are inconsistent.

13. An electronic device comprising: at least one processor; as well as A memory coupled to the at least one processor and having instructions stored thereon, the instructions, when executed by the at least one processor, causing the apparatus to perform the method according to any one of claims 1-11.

14. A vehicle comprising the electronic device according to claim 13.

15. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 11.

Images