Method and device for monitoring excessive behaviors of people in vehicle

Abstract

The invention discloses a method and a device for monitoring excessive behaviors of people in a vehicle. The method comprises the following steps: acquiring sound data in a vehicle to obtain audio signal data associated with the sound data; processing the acquired audio signal data so as to identify personnel-related audio signals in the audio signal data; when the identified personnel-related audio signal reaches a corresponding threshold value, performing image acquisition on the personnel in the vehicle; and processing the acquired image of the person in the vehicle to identify the emotional state of the person in the vehicle, and judging whether the person in the vehicle has an excessive behavior or not according to the audio identification result of the related audio signal of the person and the emotional state identification result of the person in the vehicle. According to the scheme of the invention, through combination of audio recognition and image recognition of the persons in the vehicle, recognition of excessive behaviors such as noisy of the persons in the vehicle or related abnormal behaviors can be realized, so that vehicle safety and road traffic safety are improved, and potential driving safety hazards are avoided.

Description

Technical Field

[0001] The present invention relates to the field of vehicle early warning, and more particularly to a method and a device for monitoring excessive behavior of people in a vehicle. Background Art

[0002] In a moving vehicle, any two occupants (e.g., the driver and / or several passengers) may quarrel or even engage in extreme behaviors such as fighting, which may affect the driver's driving state and the vehicle's driving state, thereby causing safety problems.

[0003] Some existing solutions include proactively implementing game recommendations, persuasion and services, and photo guidance when the passengers in the car are a couple based on image recognition. Some existing solutions include only collecting sound information in the car to determine whether there is abnormal behavior in the car.

[0004] However, these methods cannot provide accurate judgment and warning suitable for more types of excessive behaviors of occupants in the vehicle, nor can they provide more suitable interactive instructions to improve the safety status of the vehicle.

[0005] Therefore, there is a need for an improved solution for monitoring the excessive behavior of people in a vehicle. Summary of the invention

[0006] In order to solve at least one of the above technical problems, the present invention provides a method and device for monitoring excessive behavior of people in a vehicle.

[0007] According to a first aspect of the present invention, a method for monitoring excessive behavior of a person in a vehicle is provided, the method comprising:

[0008] Collecting in-vehicle sound data to obtain audio signal data associated with the sound data;

[0009] Processing the acquired audio signal data to identify the person-related audio signal therein;

[0010] When the identified person-related audio signal reaches a corresponding threshold, capturing an image of the person in the vehicle;

[0011] Processing the captured images of the occupants of the vehicle to identify the emotional state of the occupants; and

[0012] According to the audio recognition result of the person-related audio signal and the emotional state recognition result of the person in the car, it is determined whether the person in the car has excessive behavior.

[0013] In one embodiment, the method further includes: when it is determined that the person in the vehicle has engaged in excessive behavior, obtaining the position of the person in the vehicle who has engaged in excessive behavior and current vehicle status information; generating interactive instructions based on the obtained position of the person in the vehicle who has engaged in excessive behavior and current vehicle status information, and sending corresponding reminders and / or control information to the person in the vehicle who has engaged in excessive behavior and / or the current vehicle.

[0014] In one embodiment, the processing of the acquired audio signal data to identify the person-related audio signal therein includes: dividing the audio signal data of a fixed time length into at least one audio signal segment data of a specified acquisition frequency; performing audio classification on each of the at least one audio signal segment data through a convolutional neural network model, and determining whether the audio classification result of each of the at least one audio signal segment data is human speaking audio.

[0015] In one embodiment, the processing of the acquired audio signal data to identify the person-related audio signal therein further includes: determining the ratio of human voice based on the audio classification results of all audio signal segment data of the acquired audio signal; when the ratio of human voice exceeds the corresponding preset threshold, determining the audio intensity value of the person-related audio signal in the acquired audio signal data; and using an audio intensity discrimination model to calculate the human voice intensity level of the person-related audio signal based on the determined audio intensity value of the person-related audio signal.

[0016] In one embodiment, when the identified person-related audio signal reaches a corresponding threshold, capturing images of the people in the vehicle includes: when the human voice intensity level is greater than the sound intensity threshold, judging that the people in the vehicle may have excessive behavior, and capturing images of the people in the vehicle in response thereto, wherein the captured image information data includes at least the facial expressions, body movements and positions of the people in the vehicle.

[0017] In one embodiment, the current vehicle status information includes the speed, driving direction or driving position of the current vehicle, wherein generating interactive instructions based on the acquired position of the person with excessive behavior in the vehicle and the current vehicle status information, and sending corresponding reminders and / or control information to the person in the vehicle with excessive behavior and / or the current vehicle further includes: sending early warning sound reminder information and / or information to control the current vehicle driving status to the person with excessive behavior.

[0018] In one embodiment, the information of sending a warning sound reminder to a person who has excessive behavior and / or controlling the current driving state of the vehicle includes:

[0019] When it is determined that one of the persons with excessive behavior is a driver according to the position of the person with excessive behavior in the vehicle, when the speed of the current vehicle is greater than the first speed, a warning sound reminder message is issued to the driver and passengers with excessive behavior, and a control message is sent to the current vehicle to make it enter a slow deceleration mode;

[0020] When it is determined that one of the persons who have committed the excessive behavior is the driver according to the position of the person who has committed the excessive behavior in the vehicle, when the current speed of the vehicle is lower than the first speed, a warning sound reminder message is issued to the driver and passengers who have committed the excessive behavior;

[0021] When it is determined based on the positions of the persons with excessive behavior in the vehicle that the persons with excessive behavior are all passengers, when the current speed of the vehicle is greater than a second speed, which is higher than the first speed, a warning sound reminder message is issued to the driver and the passengers with excessive behavior; and

[0022] When it is determined based on the positions of the persons engaging in excessive behavior in the vehicle that the persons engaging in excessive behavior are all passengers, when the current speed of the vehicle is lower than the second speed, an early warning sound reminder message is issued to the passengers engaging in excessive behavior.

[0023] According to a second aspect of the present invention, there is provided a device for monitoring excessive behavior of a person in a vehicle, the device comprising:

[0024] An audio acquisition module, wherein the audio acquisition module is configured to collect in-vehicle sound data to obtain audio signal data associated with the sound data;

[0025] An audio recognition module, wherein the audio recognition module is configured to process the acquired audio signal data to identify a person-related audio signal therein;

[0026] An image acquisition module, wherein the image acquisition module is configured to acquire an image of a person in the vehicle when the identified person-related audio signal reaches a corresponding threshold;

[0027] an image recognition module, the image recognition module being configured to process the collected image of the occupant of the vehicle to recognize the emotional state of the occupant of the vehicle; and

[0028] An excessive behavior judgment module is configured to judge whether the person in the vehicle has excessive behavior based on the audio recognition result of the person-related audio signal and the emotional state recognition result of the person in the vehicle.

[0029] In one embodiment, the device further includes a command interaction module, which is configured to: when it is determined that the person in the vehicle has engaged in excessive behavior, obtain the position of the person in the vehicle who has engaged in excessive behavior and current vehicle status information; and generate interaction instructions based on the acquired position of the person in the vehicle who has engaged in excessive behavior and current vehicle status information, and send corresponding reminders and / or control information to the person in the vehicle who has engaged in excessive behavior and / or the current vehicle.

[0030] In one embodiment, the audio recognition module is further configured to: divide audio signal data of a fixed time length into at least one audio signal segment data of a specified acquisition frequency; perform audio classification on each of the at least one audio signal segment data through a convolutional neural network model, and determine whether the audio classification result of each of the at least one audio signal segment data is human speaking audio.

[0031] In one embodiment, the audio recognition module is further configured to: determine the ratio of human voice based on the audio classification results of all audio signal segment data of the acquired audio signal; when the ratio of human voice exceeds the corresponding preset threshold, determine the audio intensity value of the person-related audio signal in the acquired audio signal data; and use the audio intensity discrimination model to calculate the human voice intensity level of the person-related audio signal based on the determined audio intensity value of the person-related audio signal.

[0032] In one embodiment, the image acquisition module is further configured to: when the human voice intensity level is greater than the sound intensity threshold, determine that the occupant of the vehicle may have engaged in excessive behavior, and in response thereto, capture images of the occupant of the vehicle, wherein the captured image information data includes at least the facial expressions, body movements, and position of the occupant of the vehicle.

[0033] In one embodiment, the current vehicle status information includes the current vehicle speed, driving direction or driving position, wherein the instruction interaction module is further configured to: send a warning sound reminder message to a person who has excessive behavior and / or control the current vehicle driving status information.

[0034] In one embodiment, the information of sending a warning sound reminder to a person who has excessive behavior and / or controlling the current driving state of the vehicle includes:

[0035] When it is determined that one of the persons with excessive behavior is a driver according to the position of the person with excessive behavior in the vehicle, when the speed of the current vehicle is greater than the first speed, a warning sound reminder message is issued to the driver and passengers with excessive behavior, and a control message is sent to the current vehicle to make it enter a slow deceleration mode;

[0036] When it is determined that one of the persons who have committed the excessive behavior is the driver according to the position of the person who has committed the excessive behavior in the vehicle, when the current speed of the vehicle is lower than the first speed, a warning sound reminder message is issued to the driver and passengers who have committed the excessive behavior;

[0037] When it is determined based on the positions of the persons with excessive behavior in the vehicle that the persons with excessive behavior are all passengers, when the current speed of the vehicle is greater than a second speed, which is higher than the first speed, a warning sound reminder message is issued to the driver and the passengers with excessive behavior; and

[0038] When it is determined based on the positions of the persons engaging in excessive behavior in the vehicle that the persons engaging in excessive behavior are all passengers, when the current speed of the vehicle is lower than the second speed, an early warning sound reminder message is issued to the passengers engaging in excessive behavior.

[0039] According to a third aspect of the present invention, there is provided a computer device comprising a memory and a processor, wherein the memory stores computer instructions, and wherein when the computer instructions are executed by the processor, the method for monitoring excessive behavior of people in a vehicle is executed.

[0040] According to a fourth aspect of the present invention, there is provided a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for monitoring the aggressive behavior of people in a vehicle is executed.

[0041] By adopting the above technical solution, the method and device for monitoring the excessive behavior of people in a vehicle of the present invention have at least the following advantages:

[0042] Through the method for monitoring the excessive behavior of people in the car, the sound data and image data of the people in the car are collected, processed and identified, and then combined with the audio recognition results and image recognition results of the people in the car, a more accurate judgment of whether the people in the car have excessive behavior can be achieved. In addition, when judging that the people in the car have excessive behavior, different interactive instructions can be further generated in combination with the vehicle status information to more appropriately and effectively send corresponding reminders and / or control information to the people in the car and / or the current vehicle. In summary, the scheme of the present invention can improve the accuracy of judging excessive behavior with the help of the above method, thereby improving vehicle safety and road traffic safety, and avoiding potential driving safety hazards. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Non-limiting and non-exhaustive embodiments of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified.

[0044] Figure 1The flowchart of a method for monitoring aggressive behavior of a person in a vehicle according to an embodiment of the present invention is illustrated.

[0045] Figure 2 The flowchart of a method for monitoring aggressive behavior of a person in a vehicle according to another embodiment of the present invention is illustrated.

[0046] Figure 3 is based on Figure 1 or Figure 2 A schematic flow chart of an embodiment of a method for monitoring excessive behavior of people in a vehicle.

[0047] Figure 4 is based on Figure 1 or Figure 2 A schematic flow chart of another embodiment of a method for monitoring excessive behavior of people in a vehicle.

[0048] Figure 5 A block diagram of an apparatus 500 for monitoring excessive behavior of a person in a vehicle according to an embodiment of the present invention is illustrated.

[0049] In all several views of the accompanying drawings, corresponding reference characters indicate corresponding parts. It will be appreciated by the skilled person that the elements in the accompanying drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, the sizes of some elements in the accompanying drawings may be exaggerated relative to other elements to help improve the understanding of various embodiments of the present invention. In addition, common but easily understood elements that are useful or necessary in commercially feasible embodiments are generally not depicted so as to less hinder the viewing of these various embodiments of the present invention. DETAILED DESCRIPTION

[0050] In the following description, many specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that specific details need not be employed to practice the present invention. In other cases, well-known materials or methods are not described in detail to avoid obscuring the present invention.

[0051] References throughout this specification to "one embodiment", "another embodiment", "an example", or "an example", "another example" mean that the specific features, structures, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment", "in one embodiment", "in another embodiment", "an example", "an example", or "another example" appearing in various places throughout this specification do not necessarily all refer to the same embodiment or example. In addition, the specific features, structures, or characteristics may be combined in any suitable combinations and / or sub-combinations in one or more embodiments or examples. In addition, it should be understood that the drawings provided herewith are for the purpose of explanation to those of ordinary skill in the art and that the drawings are not necessarily drawn to scale.

[0052] Figure 1 The flowchart of the method 100 for monitoring the excessive behavior of the occupants of the vehicle according to one embodiment of the present invention is illustrated. The method 100 includes step S102, step S104, step S106, step S108 and step S110. The method 100 can identify whether the occupants of the vehicle have excessive behavior. The excessive behavior of the occupants of the vehicle generally refers to quarrels, fights, loud noises or other related abnormal behaviors that affect the safe driving of the vehicle. Specifically, the occupants of the vehicle generally include but are not limited to the driver and the passengers, and the excessive behavior of the occupants of the vehicle includes but is not limited to the excessive behavior such as quarrels between the rear passengers, the excessive behavior such as quarrels between the co-pilot and the rear passengers, the excessive behavior such as quarrels between the driver and the co-pilot, the excessive behavior between the driver and the rear passengers, or the excessive behavior of a single person among the driver and the passengers, etc.

[0053] In step S102, when the vehicle starts, an audio collection module such as an audio sensor or other similar sound collection devices may be used to collect in-vehicle sound data to obtain audio signal data associated with the sound data. Specifically, in-vehicle sounds generally include sounds from people talking or arguing, sounds from playing music or broadcasting, sounds from objects falling or hitting, or other sounds that can be heard inside the vehicle.

[0054] In step S104, an audio recognition module may be used to process the acquired audio signal data to recognize human-related audio signals therein.

[0055] In step S106, when the identified person-related audio signal reaches a corresponding threshold, it can be preliminarily determined that the person in the vehicle may have excessive behavior, thereby triggering the use of an image acquisition module such as an image sensor to capture images of the person in the vehicle.

[0056] In step S108, an image recognition module may be used to process the collected images of the occupants of the vehicle to identify the emotional state of the occupants of the vehicle.

[0057] In step S110, an aggressive behavior judgment module may be used to judge whether the occupant has engaged in aggressive behavior based on the audio recognition result of the occupant-related audio signal and the emotional state recognition result of the occupant.

[0058] In the above implementation scheme, advantageously, by combining audio recognition and image recognition for the people in the car, more accurate recognition of excessive behaviors such as quarrels or related abnormal behaviors of people in the car can be achieved. Moreover, only when the audio recognition module recognizes that the relevant audio signal of the person reaches the corresponding threshold, it is preliminarily judged that the people in the car may have excessive behavior, and then the image acquisition module is triggered to collect images of the people in the car. The advantage of doing so is that it can not only reduce the workload of image acquisition, but also improve the efficiency and accuracy of image recognition, thereby improving the efficiency and accuracy of identifying excessive behaviors of people in the car.

[0059] Figure 2 The flowchart of a method 200 for monitoring the aggressive behavior of a person in a vehicle according to another embodiment of the present invention is illustrated. In addition to steps S102, S104, S106, S108 and S110 in the method 100 described above, the method 200 further includes steps S212 and S214.

[0060] In step S212, when it is determined that the person in the vehicle has an aggressive behavior, the position of the person in the vehicle who has the aggressive behavior and the current vehicle state information are obtained. As described above with respect to the person in the vehicle, it should be understood that the position of the person in the vehicle can be, for example, the driver's position and the passenger's position (co-pilot position or back seat).

[0061] In step S214, an interaction instruction is generated based on the acquired position of the person with excessive behavior in the vehicle and the current vehicle status information, and corresponding reminders and / or control information are sent to the person in the vehicle with excessive behavior and / or the current vehicle.

[0062] In this embodiment, advantageously, interaction instructions can be generated in combination with the location of the person engaging in excessive behavior in the vehicle and the vehicle status information, and the current vehicle and / or the person in the vehicle are controlled to interact with the current vehicle based on the generated interaction instructions, so as to achieve a more targeted limitation on the vehicle interaction mode after the excessive behavior of the person in the vehicle is identified, and then corresponding countermeasures can be taken more accurately to avoid potential driving safety hazards in an efficient and targeted manner.

[0063] Figure 3 is based on Figure 1 or Figure 2 A schematic flowchart 300 of an embodiment of a method for monitoring excessive behavior of a person in a vehicle is provided. The flowchart 300 shows that step S104 further includes step S3041 and step S3043.

[0064] In step S3041, the audio signal data of a fixed time length is divided into at least one audio signal segment data of a specified acquisition frequency.

[0065] In step S3043, each of the at least one audio signal segment data is audio classified through a convolutional neural network model to determine whether the audio classification result of each of the at least one audio signal segment data is human speech audio.

[0066] Specifically, as described above regarding the included in-vehicle sounds, it should be understood that the audio classification results may include but are not limited to the following types: human speech audio, small pet barking audio, music or broadcast audio, collision sound audio, or other sound audio that can be heard inside the vehicle. In addition, audio signal data associated with the sound data and the corresponding audio classification results may be stored in a deep learning-based convolutional neural network model or other preset database. For example, each of at least one audio signal segment data segmented from audio signal data of a fixed time length may be audio classified through a convolutional neural network model to determine which of the audio classification results of each of the at least one audio signal segment data is human speech audio.

[0067] In this embodiment, advantageously, by classifying the audio signal data to obtain relevant audio classification results, human speech audio is screened out, and noise data is reduced to more accurately identify the presence of excessive behaviors such as quarrels among people in the vehicle.

[0068] Figure 4 is based on Figure 1 or Figure 2 A schematic flowchart 400 of another embodiment of a method for monitoring excessive behavior of a person in a vehicle. The flowchart 400 includes Figure 3 In addition to step S3041 and step S3043, the method also includes step S4045, step S4047 and step S4049.

[0069] In step S4045, the ratio of human voice is determined according to the audio classification results of all the audio signal segment data of the acquired audio signal. The ratio of human voice can be expressed as the ratio of the number of human speech audio for which the audio classification result of each of the at least one audio signal segment data is human speech audio to the total number of all the audio signal segment data or their audio classification results within the fixed time length.

[0070] In step S4047, when the ratio of the human voice exceeds the corresponding preset threshold, the audio intensity value of the person-related audio signal in the acquired audio signal data is determined.

[0071] In step S4049, the audio intensity discrimination model is used to calculate the voice intensity level of the person-related audio signal according to the determined audio intensity value of the person-related audio signal.

[0072] Specifically, in one embodiment, when the ratio of the human voice exceeds the corresponding preset threshold, it can be determined that the acquired audio signal data with a fixed time length is human speech audio, and the audio intensity value of the person-related audio signal in the acquired audio signal data is determined (for example, the audio intensity value of the person-related audio signal of all audio signal segment data or the audio intensity value of the person-related audio signal in one or more audio signal segment data can be determined). Then, the human voice intensity level of the person-related audio signal is calculated based on the determined audio intensity value through the audio intensity discrimination model. In addition, optionally, the person-related audio signal can be further processed to determine the number of people who engage in extreme behaviors such as quarrels.

[0073] In this embodiment, advantageously, by analyzing the proportion of audio signal segment data whose audio classification result is human speech audio in all audio signal segment data, it is possible to more accurately determine whether the acquired audio signal segment data with a fixed time length is human speech audio, further reduce noise interference, and more accurately identify the presence of excessive behaviors such as quarrels among people in the car.

[0074] In one embodiment, reference Figure 1-Figure 4 The described method, based on step S4049, when the identified person-related audio signal reaches the corresponding threshold, the image capture of the person in the car (step 106) also includes: when the human voice intensity level is greater than the sound intensity threshold, it is determined that the person in the car may have an extreme behavior, and in response, the image of the person in the car is captured, wherein the captured image information data at least includes the facial expression, body movement and position of the person in the car. Preferably, the position information of the person who has an extreme behavior is determined in combination with the audio signal data and the captured image information data. As described above, the person who has an extreme behavior mainly includes: multiple rear-seat personnel, co-pilot and rear-seat passengers, driver and co-pilot, driver and rear-seat passengers, or a single person among the driver and passengers.

[0075] Then proceed to step S108, process the collected images of the occupants in the vehicle to identify the emotional state of the occupants in the vehicle. For example, the collected image data is identified to identify the number of occupants and their expressions and whether the occupants and / or the driver are angry or furious, and the number of people who have engaged in extreme behaviors such as quarreling can be determined.

[0076] In one embodiment, the current vehicle status information includes the speed, driving direction or driving position of the current vehicle, wherein the generating of interactive instructions based on the acquired position of the person with excessive behavior in the vehicle and the current vehicle status information, and sending corresponding reminders and / or control information to the person in the vehicle with excessive behavior and / or the current vehicle (step S214) further includes: sending warning sound reminder information and / or information controlling the current vehicle driving state to the person with excessive behavior. For example, sending warning sound reminder information may include voice broadcast, music playback, alarm, sending out a help signal, and other forms or any combination thereof; information controlling the current vehicle driving state may include but is not limited to controlling the current vehicle's driving speed, driving direction, automatic driving takeover, or other driving control information.

[0077] In one embodiment, the information of sending a warning sound reminder to a person who has excessive behavior and / or controlling the current driving state of the vehicle includes:

[0078] When it is determined that one of the persons with excessive behavior is a driver according to the position of the person with excessive behavior in the vehicle, when the speed of the current vehicle is greater than the first speed, a warning sound reminder message is issued to the driver and passengers with excessive behavior, and a control message is sent to the current vehicle to make it enter a slow deceleration mode;

[0079] When it is determined that one of the persons who have committed the excessive behavior is the driver according to the position of the person who has committed the excessive behavior in the vehicle, when the current speed of the vehicle is lower than the first speed, a warning sound reminder message is issued to the driver and passengers who have committed the excessive behavior;

[0080] When it is determined based on the positions of the persons with excessive behavior in the vehicle that the persons with excessive behavior are all passengers, when the current speed of the vehicle is greater than a second speed, which is higher than the first speed, a warning sound reminder message is issued to the driver and the passengers with excessive behavior; and

[0081] When it is determined based on the positions of the persons engaging in excessive behavior in the vehicle that the persons engaging in excessive behavior are all passengers, when the current speed of the vehicle is lower than the second speed, an early warning sound reminder message is issued to the passengers engaging in excessive behavior.

[0082] Specifically, referring to Table 1 as follows, as shown in Table 1, the first speed may be selected as 60 km / h and the second speed may be selected as 80 km / h.

[0083] Table 1

[0084]

[0085] In this implementation, interactive instructions are generated by combining the audio recognition results and image recognition results of the occupants of the vehicle with the vehicle status information, and corresponding countermeasures are accurately taken against the excessive behaviors of the occupants of the vehicle, thereby improving safe driving behavior and road traffic safety and avoiding potential driving safety hazards.

[0086] Figure 5 The block diagram of a device 500 for monitoring the excessive behavior of people in a car according to an embodiment of the present invention is illustrated. The device 500 includes at least an audio acquisition module 502, an audio recognition module 504, an image acquisition module 506, an image recognition module 508 and an excessive behavior judgment module 510. The audio acquisition module 502 is configured to collect sound data in the car to obtain audio signal data associated with the sound data. The audio recognition module 504 is configured to process the acquired audio signal data to identify the person-related audio signal therein. The image acquisition module 506 is configured to collect images of the people in the car when the identified person-related audio signal reaches a corresponding threshold. The image recognition module 508 is configured to process the collected images of the people in the car to identify the emotional state of the people in the car. The excessive behavior judgment module 510 is configured to judge whether the people in the car have excessive behavior based on the audio recognition results of the person-related audio signal and the emotional state recognition results of the people in the car.

[0087] The audio acquisition module 502 and the image acquisition module 506 can be implemented by means of an audio sensor or camera installed on the vehicle, or an audio sensor or camera installed on the vehicle user's smart mobile terminal (e.g., a smart phone, a watch, or other wearable device). The audio recognition module 504, the image recognition module 508, and / or the excessive behavior judgment module 510 can be set on a vehicle-mounted mobile terminal, a vehicle user's smart mobile terminal, a remote server, or a dedicated processor.

[0088] In one embodiment, reference Figure 5The device 500 further includes a command interaction module 511, which can at least obtain information related to the vehicle status. The command interaction module 511 is configured to: when it is determined that the person in the vehicle has excessive behavior, obtain the position of the person in the vehicle with excessive behavior and the current vehicle status information, and generate an interaction command based on the obtained position of the person in the vehicle with excessive behavior and the current vehicle status information, and send corresponding reminders and / or control information to the person in the vehicle with excessive behavior and / or the current vehicle.

[0089] In addition, the device 500 may also include an instruction input and output interface module 515, which is configured to send the generated interactive instructions to the occupants of the vehicle with excessive behavior and / or the current vehicle, or is configured to receive interactive instructions related to the occupants of the vehicle with excessive behavior and / or the current vehicle.

[0090] In one embodiment, the audio recognition module 504 is further configured to: divide the audio signal data of a fixed time length into at least one audio signal segment data of a specified acquisition frequency, perform audio classification on each of the at least one audio signal segment data through a convolutional neural network model, and determine whether the audio classification result of each of the at least one audio signal segment data is human speaking audio.

[0091] In one embodiment, the audio recognition module 504 is further configured to: determine the ratio of human voice based on the audio classification results of all audio signal segment data of the acquired audio signal; when the ratio of human voice exceeds the corresponding preset threshold, determine the audio intensity value of the person-related audio signal in the acquired audio signal data; and use the audio intensity discrimination model to calculate the human voice intensity level of the person-related audio signal based on the determined audio intensity value of the person-related audio signal.

[0092] In one embodiment, the image acquisition module 506 is further configured to: when the human voice intensity level is greater than the sound intensity threshold, determine that the occupants in the vehicle may have engaged in excessive behavior, and in response thereto, capture images of the occupants in the vehicle, wherein the captured image information data includes at least the facial expressions, body movements, and positions of the occupants in the vehicle.

[0093] In one embodiment, the instruction interaction module 511 is further configured to send corresponding reminder information to the person who has excessive behavior according to the position of the person who has excessive behavior in the vehicle and the current speed of the vehicle, and send control information to the vehicle when necessary, as described above. Of course, as those skilled in the art will think of after reading this article, it is also possible to provide targeted reminders to vehicle users or provide more effective and timely control of the vehicle, such as switching to automatic driving mode or driving assistance, based on the specific type of excessive behavior or other status information of the current vehicle. These similar improvements should be covered within the scope of protection of this patent.

[0094] The device for monitoring excessive behavior of people in a vehicle herein can implement the method for monitoring excessive behavior of people in a vehicle discussed in any of the above implementation schemes or examples, and make similar extensions, which will not be repeated here.

[0095] Another aspect of the present invention provides a computer device, which includes a memory and a processor, wherein the memory stores computer instructions, characterized in that the computer instructions, when executed by the processor, cause the method for monitoring the excessive behavior of people in the vehicle described above to be executed. The computer device can be broadly a server, a vehicle-mounted terminal, or any other electronic device with necessary computing and / or processing capabilities. In one embodiment, the computer device may include a processor, a memory, a network interface, a communication interface, etc. connected via a system bus. The processor of the computer device can be used to provide necessary computing, processing and / or control capabilities. The memory of the computer device may include a non-volatile storage medium and an internal memory. An operating system, a computer program, etc. may be stored in or on the non-volatile storage medium. The internal memory can provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the computer device can be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of any of the above-mentioned methods for monitoring the excessive behavior of people in the vehicle of the present invention can be implemented.

[0096] Another aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for monitoring the excessive behavior of people in a vehicle as described in any of the above embodiments is implemented.

[0097] Those of ordinary skill in the art will appreciate that all or part of the steps of the above-mentioned method for monitoring the excessive behavior of people in the vehicle can be completed by instructing related hardware such as computer equipment or processors through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium, and when the computer program is executed, the steps of the method for monitoring the excessive behavior of people in the vehicle of the present invention are executed. Depending on the circumstances, any reference to memory, storage, database or other media herein may include non-volatile and / or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0098] The above description of the illustrated embodiments of the present invention, including what is described in the Abstract, is not intended to be exhaustive or limiting to the exact form disclosed. Although specific embodiments and embodiments of the present invention are described herein for illustrative purposes, various equivalent modifications are possible without departing from the broader spirit and scope of the present invention.

Claims

1. A method for monitoring the excessive behavior of people in a vehicle, characterized in that: The method comprises: Collecting in-vehicle sound data to obtain audio signal data associated with the sound data, The acquired audio signal data is processed to identify the person-related audio signal therein, When the identified person-related audio signal reaches a corresponding threshold, an image of the person in the vehicle is captured. Processing the captured images of the occupants of the vehicle to identify the emotional state of the occupants, and According to the audio recognition result of the person-related audio signal and the emotional state recognition result of the person in the car, it is determined whether the person in the car has excessive behavior.

2. The method according to claim 1, characterized in that The method further comprises: When it is determined that the person in the vehicle has an aggressive behavior, the position of the person in the vehicle and the current vehicle status information are obtained. An interactive instruction is generated based on the acquired position of the person with excessive behavior in the vehicle and the current vehicle status information, and corresponding reminder and / or control information is sent to the person in the vehicle with excessive behavior and / or the current vehicle.

3. The method according to claim 1 or 2, characterized in that: The step of processing the acquired audio signal data to identify the person-related audio signal therein comprises: The audio signal data of a fixed time length is divided into at least one audio signal segment data of a specified acquisition frequency, Perform audio classification on each of the at least one audio signal segment data through a convolutional neural network model, and determine whether the audio classification result of each of the at least one audio signal segment data is human speech audio.

4. The method according to claim 3, characterized in that The step of processing the acquired audio signal data to identify the person-related audio signal therein further includes: Determine the ratio of human voice according to the audio classification results of all audio signal segment data of the acquired audio signal, When the ratio of the human voice exceeds the corresponding preset threshold, determining the audio intensity value of the person-related audio signal in the acquired audio signal data, and The audio intensity discrimination model is used to calculate the human voice intensity level of the human-related audio signal according to the determined audio intensity value of the human-related audio signal.

5. The method according to claim 4, characterized in that When the identified person-related audio signal reaches a corresponding threshold, capturing an image of the person in the vehicle includes: When the human voice intensity level is greater than the sound intensity threshold, it is determined that the occupant of the vehicle may have engaged in excessive behavior, and in response thereto, an image of the occupant of the vehicle is captured, wherein the captured image information data includes at least the facial expressions, body movements, and position of the occupant of the vehicle.

6. The method according to claim 2, characterized in that The current vehicle status information includes the speed, driving direction or driving position of the current vehicle, wherein generating interactive instructions based on the acquired position of the person with excessive behavior in the vehicle and the current vehicle status information, and sending corresponding reminders and / or control information to the person in the vehicle with excessive behavior and / or the current vehicle further includes: sending early warning sound reminder information and / or information on controlling the current vehicle's driving status to the person with excessive behavior.

7. The method according to claim 6, characterized in that The information of sending warning sound reminder information to the person who has excessive behavior and / or controlling the current driving state of the vehicle includes: When it is determined that one of the persons with excessive behavior is a driver according to the position of the person with excessive behavior in the vehicle, when the speed of the current vehicle is greater than the first speed, a warning sound reminder message is issued to the driver and passengers with excessive behavior, and a control message is sent to the current vehicle to make it enter a slow deceleration mode; When it is determined that one of the persons who have committed the excessive behavior is the driver according to the position of the person who has committed the excessive behavior in the vehicle, when the current speed of the vehicle is lower than the first speed, a warning sound reminder message is issued to the driver and passengers who have committed the excessive behavior; When it is determined based on the positions of the persons with excessive behavior in the vehicle that the persons with excessive behavior are all passengers, when the current speed of the vehicle is greater than a second speed, which is higher than the first speed, a warning sound reminder message is issued to the driver and the passengers with excessive behavior; and When it is determined based on the positions of the persons engaging in excessive behavior in the vehicle that the persons engaging in excessive behavior are all passengers, when the current speed of the vehicle is lower than the second speed, an early warning sound reminder message is issued to the passengers engaging in excessive behavior.

8. A device for monitoring the excessive behavior of people in a vehicle, characterized in that: The device comprises: an audio acquisition module, wherein the audio acquisition module is configured to collect in-vehicle sound data to obtain audio signal data associated with the sound data, an audio recognition module, wherein the audio recognition module is configured to process the acquired audio signal data to identify the person-related audio signal therein, An image acquisition module, wherein the image acquisition module is configured to acquire an image of the person in the vehicle when the identified person-related audio signal reaches a corresponding threshold value, an image recognition module, the image recognition module being configured to process the collected images of the occupants of the vehicle to recognize the emotional state of the occupants, and An excessive behavior judgment module is configured to judge whether the person in the car has excessive behavior based on the audio recognition result of the person-related audio signal and the emotional state recognition result of the person in the car.

9. The device according to claim 8, characterized in that The device further includes an instruction interaction module, which is configured to: When it is determined that the person in the vehicle has an aggressive behavior, the position of the person in the vehicle and the current vehicle status information are obtained, and An interactive instruction is generated based on the acquired position of the person with excessive behavior in the vehicle and the current vehicle status information, and corresponding reminder and / or control information is sent to the person in the vehicle with excessive behavior and / or the current vehicle.

10. The device according to claim 8 or 9, characterized in that The audio recognition module is further configured to: The audio signal data of a fixed time length is divided into at least one audio signal segment data of a specified acquisition frequency, Perform audio classification on each of the at least one audio signal segment data through a convolutional neural network model, and determine whether the audio classification result of each of the at least one audio signal segment data is human speech audio.

11. The device according to claim 10, characterized in that The audio recognition module is further configured to: Determine the ratio of human voice according to the audio classification results of all audio signal segment data of the acquired audio signal, When the ratio of the human voice exceeds the corresponding preset threshold, determining the audio intensity value of the person-related audio signal in the acquired audio signal data, and The audio intensity discrimination model is used to calculate the human voice intensity level of the human-related audio signal according to the determined audio intensity value of the human-related audio signal.

12. The device according to claim 11, characterized in that The image acquisition module is further configured as follows: When the human voice intensity level is greater than the sound intensity threshold, it is determined that the occupant of the vehicle may have engaged in excessive behavior, and in response thereto, images of the occupant of the vehicle are captured, wherein the captured image information data includes at least the facial expressions, body movements, and position of the occupant of the vehicle.

13. The device according to claim 9, characterized in that The current vehicle status information includes the current vehicle speed, driving direction or driving position, wherein the instruction interaction module is further configured to: send a warning sound reminder message to the person who has excessive behavior and / or control the current vehicle driving status information.

14. The device according to claim 13, characterized in that The information of sending warning sound reminder information to the person who has excessive behavior and / or controlling the current driving state of the vehicle includes: When it is determined that one of the persons with excessive behavior is a driver according to the position of the person with excessive behavior in the vehicle, when the speed of the current vehicle is greater than the first speed, a warning sound reminder message is issued to the driver and passengers with excessive behavior, and a control message is sent to the current vehicle to make it enter a slow deceleration mode; When it is determined that one of the persons who have committed the excessive behavior is the driver according to the position of the person who has committed the excessive behavior in the vehicle, when the current speed of the vehicle is lower than the first speed, a warning sound reminder message is issued to the driver and passengers who have committed the excessive behavior; When it is determined based on the positions of the persons with excessive behavior in the vehicle that the persons with excessive behavior are all passengers, when the current speed of the vehicle is greater than a second speed, which is higher than the first speed, a warning sound reminder message is issued to the driver and the passengers with excessive behavior; and When it is determined based on the positions of the persons engaging in excessive behavior in the vehicle that the persons engaging in excessive behavior are all passengers, when the current speed of the vehicle is lower than the second speed, an early warning sound reminder message is issued to the passengers engaging in excessive behavior.

15. A computer device comprising a memory and a processor, wherein the memory stores computer instructions, characterized in that: When the computer instructions are executed by the processor, the method for monitoring excessive behavior of a person in a vehicle according to any one of claims 1 to 7 is executed.

16. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for monitoring excessive behavior of a person in a vehicle according to any one of claims 1 to 7 is executed.

Images