Intelligent processing method and device for behavior information of rear-row users of cockpit and storage medium

Abstract

The invention provides an intelligent processing method and device for behavior information of users in the rear row of a cockpit and a storage medium, and is applied to a vehicle machine system of a target vehicle, and the method comprises the steps: obtaining in-vehicle sound data of the target vehicle; if it is determined that noise exists according to the sound data, obtaining a target in-vehicle image; determining a passenger type and a current behavior of the target passenger according to the target in-vehicle image; if it is detected that the current behavior belongs to the preset reference interference behavior, the first device and / or the second device are / is controlled to execute a reminding strategy matched with the passenger type. Thus, the reminding strategy for the target passenger having the reference interference behavior in the target vehicle is executed through interaction among the multiple terminal devices, the accuracy and efficiency of determining the target passenger are improved, the diversity and practicability of the reminding strategy implementation mode are improved, and the user experience is improved. Therefore, the occurrence of driving interference behaviors is effectively reduced, and the driving safety is improved.

Description

Technical Field

[0001] This application belongs to the field of data identification technology in the new generation of information technology industry, and specifically relates to an intelligent processing method and device for user behavior information in the rear row of the cabin. Background Art

[0002] At present, with the continuous development of automobile technology and the improvement of intelligence level, the functions that can be achieved by the vehicle system are gradually enriched.

[0003] However, in a vehicle driving scenario with passengers, when rear passengers in the cabin make noise or make driving-interfering behaviors that interfere with the noise during driving, the vehicle system with voice interaction function and the driver may be affected by the noise or noise, thereby affecting the driver's driving. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, and storage medium for intelligent processing of rear-seat user behavior information in the cabin. Through interaction between multiple terminal devices, a reminder strategy is executed for target passengers with reference interference behaviors in a target vehicle, thereby improving the diversity and practicality of the reminder strategy implementation methods, thereby effectively reducing the occurrence of driving interference behaviors and improving driving safety.

[0005] In a first aspect, an embodiment of the present application provides a method for intelligently processing behavior information of rear-seat users in a cabin, which is applied to a vehicle-mounted system of a target vehicle. The method includes:

[0006] Acquiring in-vehicle sound data of the target vehicle;

[0007] If it is determined that there is noise according to the sound data, a target vehicle interior image is obtained, where the target vehicle interior image is an image of the vehicle interior at a location where a sound source corresponding to the noise is located;

[0008] determining, based on the target vehicle interior image, the passenger type and current behavior of a target passenger, the target passenger being a passenger seated in a seat corresponding to the location of the sound source;

[0009] If it is detected that the current behavior belongs to the preset reference interference behavior and the passenger type is a child, the first device is controlled to perform a first tangible reminder operation and play a first voice reminder until a closing reminder instruction sent by the second device is detected. The first tangible reminder operation includes a first vibration operation and / or a first ringing operation. The first voice reminder is used to remind the target passenger to sit down. The first device is a device worn by the target passenger, and the second device is a device held by the guardian of the target passenger.

[0010] In a second aspect, an embodiment of the present application provides an intelligent processing device for rear-seat user behavior information, which is applied to a vehicle-mounted system of a target vehicle. The device includes:

[0011] A first acquiring unit, configured to acquire in-vehicle sound data of the target vehicle;

[0012] a second acquiring unit, configured to acquire a target vehicle interior image if it is determined based on the sound data that noise exists, the target vehicle interior image being an image of the vehicle interior at a location where a sound source corresponding to the noise is located;

[0013] a determination unit, configured to determine a passenger type and current behavior of a target passenger based on the target vehicle interior image, wherein the target passenger is a passenger seated in a seat corresponding to the location of the sound source;

[0014] A control unit is configured to control a first device to perform a first sensible reminder operation and play a first voice reminder if it is detected that the current behavior belongs to a preset reference interference behavior and the passenger type is a child, until a reminder closing instruction sent by a second device is detected, wherein the first sensible reminder operation includes a first vibration operation and / or a first ringing operation, and the first voice reminder is used to remind the target passenger to sit down. The first device is a device worn by the target passenger, and the second device is a device held by the guardian of the target passenger.

[0015] In a third aspect, an embodiment of the present application provides a vehicle-mounted device comprising a processor, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the program includes instructions for executing the steps in the first aspect of the embodiment of the present application.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the steps in the first aspect of the embodiment of the present application when the computer program / instruction is executed by a processor.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements some or all of the steps described in the first aspect of the embodiment of the present application.

[0018] It can be seen that in the embodiment of the present application, after the vehicle-mounted device determines the target passenger through the sound source position and image, the vehicle-mounted device interacts with the first device of the target passenger and / or the second device of the target passenger's guardian to execute a reminder strategy to remind the target passenger to stop his or her driving-disrupting behavior, thereby improving the diversity and practicality of the reminder strategy implementation methods, providing the driver with a quiet driving environment, and effectively improving driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a structural block diagram of a behavior reminder system provided by an embodiment of the present application;

[0021] Figure 2 This is a flow chart of a method for intelligently processing behavior information of rear-seat users in a cabin, provided in an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a scenario for sound source localization inside a vehicle provided by an embodiment of the present application;

[0023] Figure 4 This is a schematic diagram of a scenario in which a first device performs a sensory reminder operation, provided in an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of a scenario in which a display screen plays a video, provided in an embodiment of the present application;

[0025] Figure 6 This is a block diagram of the functional units of an intelligent processing device for rear-seat user behavior information provided by an embodiment of the present application;

[0026] Figure 7 This is a block diagram of the functional units of another intelligent processing device for rear-seat user behavior information provided by an embodiment of the present application;

[0027] Figure 8 This is a structural block diagram of a vehicle system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0030] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] See also Figure 1 , Figure 1 This is a structural diagram of a behavior reminder system provided by an embodiment of the present application. Figure 1 As shown, the behavior reminder system 100 includes a vehicle-mounted system 110, a first device 120, and a second device 130. The first device 120 and the second device 130 are communicatively connected to the target vehicle. The first device is a device worn by the target passenger, and the second device is a device held by the target passenger's guardian. The vehicle-mounted system 110 is the abbreviation for the in-vehicle infotainment product installed in the target vehicle. The vehicle-mounted system should be able to functionally realize information communication between the person and the vehicle, and between the vehicle and the outside world (vehicle to vehicle). The target vehicle can be any model of vehicle on the market. One vehicle-mounted system 110 can simultaneously correspond to multiple first devices 120 and / or second devices 130, or the behavior reminder system 100 can include multiple vehicle-mounted systems 110, each behavior reminder system 100 corresponding to one or more first devices 120 and / or second devices 130.

[0032] Based on this, an embodiment of the present application provides an intelligent processing method for rear-seat user behavior information in the cabin. The embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0033] See also Figure 2 , Figure 2 1 is a flow chart of an intelligent processing method for rear-seat user behavior information provided in an embodiment of the present application. The method is applied to the vehicle system 110 in the behavior reminder system 100. The method further includes:

[0034] Step S201: Acquire the interior sound data of the target vehicle.

[0035] Among them, according to actual conditions, the trigger conditions for the vehicle system to obtain the in-car sound data of the target vehicle can be set. For example, when the single continuous driving distance or continuous driving time of the target vehicle exceeds the preset range, the in-car sound data will be continuously obtained. The reason is that when the target vehicle has been driving for too long, both the driver and the passengers may become irritated. At this time, the passengers are more likely to interfere with the driving and affect the driver's driving; the trigger condition can also be set to when the target vehicle starts, the vehicle system has a built-in timer to perform periodic sound data collection. It can be understood that in order to timely determine the driving interference behavior of the passengers, the interval period should not be too long, and no limitation is made here.

[0036] Step S202: If it is determined based on the sound data that there is noise, an image inside the target vehicle is acquired.

[0037] The target in-vehicle image is an in-vehicle image at the location of the sound source corresponding to the noise.

[0038] In a possible embodiment, the target vehicle includes an on-board camera and a sound collection device, and the in-vehicle sound data includes sound sub-data collected by each of the sound collection devices; obtaining the target in-vehicle image includes: performing a sound recognition operation on each of the sound sub-data in sequence to determine the target sound sub-data whose sound type is the noise; determining the location of the sound source based on the target sound sub-data; and controlling at least one of the on-board cameras corresponding to the location of the sound source to collect a preset number of the target in-vehicle images.

[0039] Among them, in order to avoid over-reliance on monitoring systems such as on-board cameras to capture images of passenger behavior, and to avoid occupying a large amount of computing resources of the vehicle system due to image processing, in this example, the sound data is first collected through the sound collection equipment inside the vehicle, and the location of the sound source corresponding to the noise is determined. The camera at the position corresponding to the sound source is then used to collect the target vehicle interior image to ensure that the vehicle system can clearly and accurately determine the actions and behaviors of the target passenger who makes the noise.

[0040] Among them, the method of determining the location of the sound source by the target sound sub-data is the sound source localization technology. Since there are multiple sound collection devices in the target vehicle, the target vehicle can be located by using a microphone array, that is, multiple microphones are used to measure the sound source at different measurement points. Since the time for the sound signal to reach different microphones has different degrees of delay (also known as time delay), the measured sound signal is processed by an algorithm to obtain the arrival direction (including azimuth, pitch angle) and distance of the sound source point relative to the microphone, and finally determine the sound source position of the sound sub-data corresponding to the noise.

[0041] For example, see Figure 3 , Figure 3 This is a schematic diagram of a scene of sound source localization inside a vehicle provided by an embodiment of the present application, such as Figure 3 As shown, the target vehicle contains three sound collection devices: microphone 1, microphone 2, and microphone 3. Microphones 1 and 2, and microphones 2 and 3, are separated by a distance d and are located on either side of the front seat. When determining the noise's acoustic sub-data, the actual distances from each microphone to the sound source are calculated based on the time difference between the sound wave signals reaching each microphone and the speed of sound. The distance from microphone 1 to the sound source is L1, the distance from microphone 2 to the sound source is L2, and the distance from microphone 3 to the sound source is L3. Using the positions of the three microphones and L1, L2, and L3, the sound source is located, namely the leftmost rear seat. The target passenger's movements can then be determined by capturing images from an onboard camera capable of capturing that location.

[0042] It can be seen that in this example, the vehicle system uses the on-board camera and sound acquisition equipment in combination, so that the vehicle system can first determine the location of the sound source through the sound data, and then call the camera at the corresponding position to obtain the image, thereby reducing the number of images that the vehicle system needs to process, improving the efficiency and accuracy of the vehicle system in determining the target passenger, and improving the stability of the vehicle system in data processing.

[0043] In a possible embodiment, the sound recognition operation is performed on each of the sound sub-data in sequence to determine the target sound sub-data whose sound type is the noise, including: determining the sound intensity, sound frequency and duration corresponding to each of the sound sub-data; if the sound intensity of the sound sub-data being processed is greater than a preset decibel, determining that the sound sub-data being processed is the sound data to be identified; if the sound intensity of the sound sub-data being processed is less than or equal to the preset decibel, determining the target frequency level based on the sound frequency and the preset frequency range set; and if the target frequency level is greater than the preset frequency level and the corresponding duration is greater than a preset number of seconds, determining that the sound sub-data being processed is the sound data to be identified; performing a speech recognition operation on the sound data to be identified to obtain a recognized text; if the recognized text represents a complete intent, determining that the sound type of the sound sub-data being processed is a conversational sound, and continuing to process the next sound sub-data until the target sound sub-data is determined; if the recognized text represents an incomplete intent, determining that the sound sub-data being processed is the target sound sub-data.

[0044] Each of the preset frequency ranges in the preset frequency range set corresponds to a frequency level.

[0045] The preset decibel level is determined based on the vehicle's environment and the driver's tolerance. When the intensity of the sound sub-data being processed exceeds the preset decibel level, the vehicle system can directly identify it as unrecognized sound data and proceed to the next step of voice recognition. Excessive sound intensity can significantly affect the driver's driving behavior. Furthermore, the target frequency level is used to characterize the sharpness of the sound sub-data. A higher frequency level indicates a sharper sound and is more likely to be identified as noise.

[0046] It can be seen that in this example, by judging the various attributes of the sound data and performing voice recognition on the sound data to distinguish whether the sound data comes from passenger conversations or machine voice of the vehicle system, the accuracy and efficiency of the vehicle system's noise determination are improved, thereby ensuring the accuracy and practicality of subsequent image acquisition operations and the execution of reminder strategies.

[0047] Step S203 : determining the passenger type and current behavior of the target passenger based on the target vehicle interior image.

[0048] The target passenger is a passenger sitting in a seat corresponding to the location of the sound source.

[0049] Among them, since the present application is aimed at reminding passengers of driving interference behaviors, it is necessary to obtain images of the target passengers when they make noises to confirm their behaviors and to select different reminder strategies according to the type of passengers.

[0050] In a possible embodiment, after determining the passenger type and current behavior of the target passenger based on the target in-vehicle image, the method further includes: if it is detected that the current behavior is abnormal behavior, controlling the first device to perform an acquisition operation to obtain the current vital signs information of the target passenger; obtaining reference vital signs information of the target passenger; determining a comparison result based on the reference vital signs information and the current vital signs information; if the comparison result indicates that the current vital signs information is in an abnormal state, controlling the second device to perform a third sensory reminder operation and play a third voice reminder until a confirmation instruction sent by the second device is received.

[0051] Among them, the abnormal behavior is the associated behavior of the disease to be monitored set in advance by the guardian through the second device, the reference vital sign information is the vital sign information collected by the first device when the target passenger is in a normal state, the third tangible reminder operation is a third vibration operation, and the third voice reminder is used to remind the guardian to pay attention to the physical condition of the target passenger.

[0052] Among them, when the target vehicle is carrying passengers with medical conditions, these passengers may exhibit some strange behaviors due to the onset of their own medical conditions, which may cause noise. For example, the target passenger may scream or hit the car door or window due to convulsions, making noise. At this time, it is necessary to interact with the first device and the second device to protect the life and health of the target passenger. Of course, it is also necessary to rule out the possibility that the target passenger may make noise by accident. Therefore, the specific analysis process is as follows: the vehicle system first detects that the target passenger has made an abnormal behavior pre-set by the guardian through the second device, then controls the first device to collect the current vital signs information of the target passenger, and then determines whether the target passenger is in an abnormal physical condition based on the reference vital signs information collected in advance, and then controls the second device to remind the guardian.

[0053] For example, see Figure 4 , Figure 4 is a schematic diagram of a scenario in which a first device performs a sensory reminder operation, as provided in an embodiment of the present application. Figure 4 As shown, when the target passenger is riding in the target vehicle, he will wear Figure 4 The 01 in the figure is the first device worn by the target passenger. The vehicle system sends a sensory reminder operation instruction to the first device, and the first device will perform the sensory reminder operation, that is, vibration operation or ringing operation. The electronic device held by the guardian of the target passenger and connected to the vehicle system is Figure 4 02 in the first device. When the first device performs the reminder operation, its display interface will show Figure 4 03 in the display shows the control selection interface. The text in the interface informs the guardian that the first device is currently performing a sensory reminder operation and whether it needs to be stopped. The guardian can then use the selection control below to choose whether to stop the operation performed by the first device. The control on the left is used to send a stop command to the vehicle system, and the control on the right is used to send a continue command to the vehicle system. The vehicle system then stops or continues the operation performed by the first device according to the command.

[0054] It can be seen that in this example, the vehicle system uses a multi-level detection process to determine whether the target passenger is indeed in an abnormal physical condition, ensuring the accuracy and efficiency of the vehicle system in determining abnormal conditions and reducing false alarms.

[0055] In a possible embodiment, the vital signs information includes multiple basic vital signs information, each of which corresponds to a sign type; determining the comparison result based on the reference vital signs information and the current vital signs information includes: determining multiple abnormal vital signs information associated with the condition to be monitored; determining the sign change corresponding to each abnormal vital signs information based on the reference vital signs information and the current vital signs information; obtaining the normal fluctuation range corresponding to each abnormal vital signs information; if each of the sign change amounts is within the corresponding normal fluctuation range, determining that the current vital signs information is in a normal state, and generating the comparison result; if there is a sign change amount that exceeds the corresponding normal fluctuation range, determining that the current vital signs information is in the abnormal state, and generating the comparison result.

[0056] The abnormal physical sign information is the abnormal basic physical sign information of the patient when the disease to be monitored occurs.

[0057] Among them, since the basic abnormal vital sign information that may appear in each monitored condition is different, by determining the related abnormal vital sign information based on the monitored condition, the priority of some basic vital sign information can be improved, and then the vehicle system can only compare the vital sign change corresponding to the abnormal vital sign information with the corresponding normal fluctuation range.

[0058] For example, when the target passenger types are elderly people and young children, this example can detect abnormal fluctuations in each basic vital sign information of the target passenger. This is because when the target passenger is a child, it is difficult for the child to accurately express his or her physical condition when feeling unwell, which may result in abnormalities not being noticed. When the target passenger is an elderly person, since the elderly person's physical condition is sub-healthy, extra attention should be paid to changes in their physical condition, so a comprehensive detection of fluctuations in vital sign information is required. Furthermore, when the change in any of the basic vital sign information exceeds the corresponding normal fluctuation range, the vehicle system will control the second device to perform a third sensory reminder operation and play a third voice reminder to attract the attention of the guardian and avoid missing the best treatment time.

[0059] It can be seen that in this example, the vehicle system determines abnormal physical sign information through the monitored condition, and only detects abnormal fluctuations in the abnormal physical sign information, thereby improving the efficiency and stability of the vehicle system in determining the abnormal physical condition of the target passenger.

[0060] Step S204: If it is detected that the current behavior belongs to the preset reference interference behavior and the passenger type is a child, the first device is controlled to perform a first sensible reminder operation and play a first voice reminder until a close reminder instruction sent by the second device is detected. The first sensible reminder operation includes a first vibration operation and / or a first ringing operation, and the first voice reminder is used to remind the target passenger to sit down.

[0061] The first device is a device worn by the target passenger, and the second device is a device held by the guardian of the target passenger.

[0062] Among them, the reference interference behavior can be determined in advance by the driver through interaction with the vehicle system, such as pulling the rear sunshade, banging the window, frequent loud noises, etc., which are not limited here.

[0063] Among them, since the second device is an electronic device held by the guardian, when the passenger type of the target passenger is a passenger type that does not have a guardian, the vehicle system can only control the first device to execute an adapted reminder strategy to remind the target passenger, reminding the target passenger to stop his driving-interfering behavior to provide the driver with a quiet driving environment, thereby improving driving safety.

[0064] In a possible embodiment, the method further includes: continuously acquiring the target vehicle interior image to detect the subsequent behavior of the target passenger; when it is detected that the subsequent behavior belongs to the reference interference behavior, controlling the first device to perform a second sensory reminder operation and play a second voice reminder; and acquiring the voice instruction of the target passenger; when the voice instruction indicates that the target passenger wants to watch the video or the cartoon, sending a play query instruction to the second device; in response to the consent to play instruction sent by the second device to the play query instruction, playing the video or the cartoon on the rear display screen corresponding to the target passenger.

[0065] The first tangible reminder operation includes a first vibration operation and / or a first ringing operation, and the first voice reminder is used to remind the target passenger to sit down. The second tangible reminder operation includes a second vibration operation and / or a second ringing operation, and the second voice reminder is used to ask the target passenger what video or cartoon he wants to watch.

[0066] Among them, the present application includes a first vibration operation, a second vibration operation, and a third vibration operation, as well as a first ringing operation and a second ringing operation. It should be noted that the vibration degree corresponding to each vibration operation is different, and the duration, volume and frequency of the ringing corresponding to each ringing operation are also different. Moreover, the volume of the sound corresponding to the ringing operation is within the acceptable range for the human body and will not affect the driver's driving, but will only attract the attention of the passenger wearing the device. It is understandable that since the control of the first device to perform the second sensible reminder operation is aimed at the target passenger's driving interference behavior again, the vibration degree of the corresponding second vibration operation will be significantly stronger than the vibration degree of the first vibration operation, and the duration, volume or frequency of the second ringing operation will be longer than the duration of the first ringing operation.

[0067] Among them, since the passenger type is a child, the video is played on the rear display screen. Because, for safety reasons, when children ride in a car, they are usually arranged to sit in the back seat. The design of the car seat is based on the body shape of adults, and children are smaller in size and cannot use the seat belt correctly when sitting in the front seat. This means that even if children wear seat belts, in the event of emergency braking or collision, they may slip out from under the seat belt or have their necks strangled by the seat belt, which brings a greater risk of injury. The back seat is more suitable for children because it is not only far away from the collision point, but also has no airbags, which can provide more safe space for children. If the passenger type is an adult, the corresponding display screen can be determined according to the actual seating position of the target passenger to play the video or cartoon, so that the passenger can watch without moving.

[0068] For example, see Figure 5 , Figure 5 This is a schematic diagram of a scenario in which a display screen plays a video provided by an embodiment of the present application, such as Figure 5 As shown, when the target passenger is a child and the front seat of the target vehicle (i.e. Figure 5 01) is equipped with a rear display screen (i.e. Figure 5 In 02), after the car system receives the child's voice command indicating that there is a cartoon and the second device sends a consent to play command, the cartoon will be played on the rear display screen. At this time, the child's emotional changes are as follows: Figure 5 As shown in Figure 03, children’s attention will be attracted by cartoons, changing from irritability to happiness. They will not engage in driving-disrupting behaviors, thus providing the driver with a quiet and comfortable driving environment.

[0069] It can be seen that in this example, for the target passenger whose passenger type is a child, the reminder strategy can be flexibly implemented through the interaction between the vehicle system and the first device and the second device, and the option of playing videos or cartoons is added to ensure that the child's attention can be attracted, thereby preventing the child from performing driving-interfering behaviors again and affecting the driver's driving of the vehicle, thereby improving the diversity and intelligence of the vehicle system's execution of reminders to the vehicle.

[0070] In a possible embodiment, the method further includes: when receiving a stop playback instruction sent by the second device, saving the video playback record according to the video playback progress; if the playback progress is less than the preset progress, updating the second voice reminder according to the video name; when receiving the voice instruction containing the video name, continuing to play the video or the cartoon according to the video playback record.

[0071] The video playback record includes the video storage location, playback progress, and video name, and the updated second voice reminder is used to remind the target passenger whether to watch the video or the cartoon corresponding to the video name.

[0072] Since the target passenger is a child, the guardian can determine whether the child should continue watching the video or cartoon based on the actual situation. The guardian can then use a second device to send a stop playback command to the vehicle system to stop the video playback operation on the corresponding rear display screen for the target passenger. To reflect the humanized design and improve the child's vehicle riding experience, after the second device sends the stop playback command, the vehicle system will save the video playback record based on the video playback progress. If the playback progress is less than the preset progress, indicating that there is still content in the video that has not been played, the vehicle system will update the second voice reminder based on the video title. The updated second voice reminder is used to remind the target passenger that they can continue watching the video or cartoon that they did not finish watching last time. If a voice command containing the video title is received later, the video playback will resume according to the saved video playback record.

[0073] As can be seen, in this example, by responding to the stop playback command and saving the video playback record, and updating the second reminder voice when the playback is not completed, the target passenger can directly choose to continue playing the video next time, which improves the intelligence of the vehicle system's video playback operation and enhances the passenger's riding experience.

[0074] visible, Figure 2This is a flow chart of an intelligent processing method for rear-seat user behavior information provided in an embodiment of the present application. After the vehicle-mounted device determines the target passenger through the sound source position and image, the vehicle-mounted device interacts with the first device of the target passenger and / or the second device of the target passenger's guardian to execute a reminder strategy to remind the target passenger to stop his or her driving-interfering behavior, thereby improving the diversity and practicality of the reminder strategy implementation methods, providing the driver with a quiet driving environment, and effectively improving driving safety.

[0075] The following is an embodiment of the device of the present application. The embodiment of the device of the present application and the embodiment of the method of the present application are based on the same concept and are used to perform the method described in the embodiment of the present application. For ease of explanation, the embodiment of the device of the present application only shows the parts related to the embodiment of the device of the present application. For specific technical details not disclosed, please refer to the description of the embodiment of the method of the present application, and will not be repeated here.

[0076] The embodiment of the present application provides an intelligent processing device for user behavior information in the rear row of the cabin. The intelligent processing device for user behavior information in the rear row of the cabin is applied to Figure 1 The vehicle-mounted system 110 in the illustrated behavior reminder system 100 further includes a first device 120 and a second device 130, which are in communication with the vehicle-mounted system 110. Specifically, the intelligent processing device for rear-seat user behavior information is configured to execute the steps described above for the vehicle-mounted system in the intelligent processing method for rear-seat user behavior information. The intelligent processing device for rear-seat user behavior information provided in this embodiment of the present application may include modules corresponding to the corresponding steps.

[0077] In the embodiments of the present application, the intelligent processing device for rear-seat user behavior information can be divided into functional modules based on the above-described method example. For example, each functional module can be divided into corresponding functional modules, or two or more functions can be integrated into a single processing module. The integrated modules can be implemented in the form of hardware or software functional modules. The module division in the embodiments of the present application is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used.

[0078] In the case of dividing each functional module into corresponding functional modules, Figure 66 is a functional unit block diagram of an intelligent processing device for rear-seat user behavior information provided by an embodiment of the present application. The intelligent processing device 60 for rear-seat user behavior information is applied to a vehicle-mounted system of a target vehicle and includes: a first acquisition unit 601 for acquiring in-vehicle sound data of the target vehicle; a second acquisition unit 602 for acquiring an in-vehicle image of the target vehicle, if noise is determined to be present based on the sound data; a determination unit 603 for determining the passenger type and current behavior of a target passenger based on the in-vehicle image; the target passenger being a passenger seated in a seat corresponding to the location of the sound source; and a control unit 604 for controlling a first device to perform a first sensory reminder operation and play a first voice reminder if it is detected that the current behavior corresponds to a preset reference interference behavior and the passenger type is a child, until a deactivation reminder instruction sent by a second device is detected. The first sensory reminder operation includes a first vibration operation and / or a first ringing operation. The first voice reminder is used to remind the target passenger to sit down. The second device is a device held by the guardian of the target passenger.

[0079] In a possible embodiment, the control unit 604 is further specifically used to: continuously acquire the target vehicle interior image to detect the subsequent behavior of the target passenger; when it is detected that the subsequent behavior belongs to the reference interference behavior, control the first device to perform a second sensible reminder operation and play a second voice reminder, the second sensible reminder operation includes a second vibration operation and / or a second ringing operation, and the second voice reminder is used to inquire about the video or cartoon that the target passenger wants to watch; and, acquire the voice instruction of the target passenger; when the voice instruction indicates that the target passenger wants to watch the video or the cartoon, send a play inquiry instruction to the second device; in response to the consent to play instruction sent by the second device for the play inquiry instruction, play the video or the cartoon on the rear display screen corresponding to the target passenger.

[0080] In a possible embodiment, the control unit 604 is further specifically used to: when receiving a stop playback instruction sent by the second device, save the video playback record according to the video playback progress, and the video playback record includes the video storage location, playback progress, and video name; if the playback progress is less than the preset progress, update the second voice reminder according to the video name, and the updated second voice reminder is used to remind the target passenger whether to watch the video or the cartoon corresponding to the video name; when receiving the voice instruction containing the video name, continue to play the video or the cartoon according to the video playback record.

[0081] In a possible embodiment, the target vehicle includes an on-board camera and a sound collection device, and the in-vehicle sound data includes sound sub-data collected by each of the sound collection devices; in terms of obtaining the target in-vehicle image, the second acquisition unit 602 is specifically used to: perform a sound recognition operation on each of the sound sub-data in sequence to determine the target sound sub-data whose sound type is the noise; determine the location of the sound source based on the target sound sub-data; and control at least one of the on-board cameras corresponding to the location of the sound source to collect a preset number of the target in-vehicle images.

[0082] In one possible embodiment, in the aspect of performing the sound recognition operation on each of the sound sub-data in sequence to determine the target sound sub-data whose sound type is the noise, the second acquisition unit 602 is specifically used to: determine the sound intensity, sound frequency and duration corresponding to each of the sound sub-data; if the sound intensity of the sound sub-data being processed is greater than a preset decibel, determine that the sound sub-data being processed is the sound data to be identified; if the sound intensity of the sound sub-data being processed is less than or equal to the preset decibel, determine the target frequency level according to the sound frequency and the preset frequency range set, and each of the preset frequency range sets is selected as the target frequency level. The preset frequency range corresponds to a frequency level; and if the target frequency level is greater than the preset frequency level and the corresponding duration is greater than the preset number of seconds, the sound sub-data being processed is determined to be the sound data to be identified; a speech recognition operation is performed on the sound data to be identified to obtain a recognized text; if the recognized text represents a complete intention, the sound type of the sound sub-data being processed is determined to be a conversational sound, and the next sound sub-data is processed until the target sound sub-data is determined; if the recognized text represents an incomplete intention, the sound sub-data being processed is determined to be the target sound sub-data.

[0083] In a possible embodiment, after determining the passenger type and current behavior of the target passenger based on the target in-vehicle image, the determination unit 603 is further specifically used to: if it is detected that the current behavior is abnormal behavior, control the first device to perform a collection operation to obtain the current vital signs information of the target passenger, where the abnormal behavior is an associated behavior of the disease to be monitored that is set in advance by the guardian through the second device; obtain reference vital signs information of the target passenger, where the reference vital signs information is the vital signs information collected by the first device when the target passenger is in a normal state; determine a comparison result based on the reference vital signs information and the current vital signs information; if the comparison result indicates that the current vital signs information is in an abnormal state, control the second device to perform a third sensible reminder operation and play a third voice reminder until a confirmation instruction sent by the second device is received, where the third sensible reminder operation is a third vibration operation, and the third voice reminder is used to remind the guardian to pay attention to the physical condition of the target passenger.

[0084] In a possible embodiment, the vital sign information includes multiple basic vital sign information, each of which corresponds to a sign type; in terms of determining the comparison result based on the reference vital sign information and the current vital sign information, the determination unit 603 is specifically used to: determine multiple abnormal vital sign information associated with the disease to be monitored, the abnormal vital sign information is abnormal basic vital sign information in the patient's body when the disease to be monitored occurs; determine the vital sign change corresponding to each abnormal vital sign information based on the reference vital sign information and the current vital sign information; obtain the normal fluctuation range corresponding to each abnormal vital sign information; if each of the vital sign change amounts is within the corresponding normal fluctuation range, determine that the current vital sign information is in a normal state, and generate the comparison result; if there is a vital sign change amount that exceeds the corresponding normal fluctuation range, determine that the current vital sign information is in the abnormal state, and generate the comparison result.

[0085] In the case of integrated units, such as Figure 7 As shown, Figure 7 This is a block diagram of the functional units of another intelligent processing device for cabin rear row user behavior information provided by an embodiment of the present application. Figure 7In the embodiment, the intelligent processing device 60 for the behavior information of users in the rear row of the cabin includes: a processing module 712 and a communication module 711. The processing module 712 is used to control and manage the actions of the intelligent processing device for the behavior information of users in the rear row of the cabin, for example, the steps of the first acquisition unit 601, the second acquisition unit 602, the determination unit 603 and the control unit 604, and / or other processes for executing the technology described herein. The communication module 711 is used to support the interaction between the intelligent processing device for the behavior information of users in the rear row of the cabin and other devices. Figure 7 As shown, the intelligent processing device for the behavior information of users in the rear seats of the cabin may include a storage module 713, which is used to store program codes and data of the intelligent processing device for the behavior information of users in the rear seats of the cabin.

[0086] The processing module 712 may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The communication module 711 may be a transceiver, an RF circuit, or a communication interface, and the like. The storage module 713 may be a memory.

[0087] Among them, all relevant contents of each scenario involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here. The above-mentioned intelligent processing device 60 for the rear-seat user behavior information can execute the above-mentioned Figure 2 The intelligent processing method of user behavior information in the back row of the cabin is shown.

[0088] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0089] Figure 8 This is a structural block diagram of a vehicle system provided by an embodiment of the present application. Figure 8 As shown, the vehicle system 110 may include one or more of the following components: a processor 810, and a memory 820 coupled to the processor 810. The memory 820 may store one or more programs 821. The one or more programs 821 may be configured to implement the methods described in the above embodiments when executed by the one or more processors 810. The vehicle system described here is the vehicle system 110 in the above embodiments.

[0090] The processor 810 may include one or more processing cores. The processor 810 utilizes various interfaces and circuits to connect various components within the vehicle-mounted system 110. It executes instructions, programs, code sets, or instruction sets stored in the memory 820, as well as accesses data stored in the memory 820, to perform various functions and process data within the vehicle-mounted system 110. Optionally, the processor 810 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 810 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 810 and may instead be implemented via a separate communications chip.

[0091] The memory 820 may include random access memory (RAM) or read-only memory (ROM). The memory 820 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 820 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described above, and the like. The data storage area may also store data created by the vehicle system 110 during use.

[0092] It is understandable that the vehicle system 110 may include more or fewer structural elements than those in the above structural block diagram, which is not limited here.

[0093] An embodiment of the present application also provides a computer storage medium, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, part or all of the steps of any method described in the above method embodiments are implemented.

[0094] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps of any method described in the above method embodiments.

[0095] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices, and systems can be implemented in other ways. For example, the device embodiments described above are merely schematic; for example, the division of the units is merely a logical function division, and there may be other division methods in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection of devices or units, which may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional units in various embodiments of the present invention may be integrated into a single processing unit, each unit may be physically included separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional units.

[0099] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a volatile memory or a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct RAM bus random access memory (DR RAM), among other media that can store program code.

[0100] Although the present invention is disclosed above, it is not limited thereto. Any person skilled in the art may readily conceive of variations or substitutions, and may make various modifications and alterations without departing from the spirit and scope of the present invention. Combinations of the above-described functions and implementation steps, including software and hardware implementations, are all within the scope of protection of the present invention.

Claims

1. An intelligent processing method for rear-seat user behavior information in a cabin, characterized by: The method is applied to a vehicle-mounted system of a target vehicle, and includes: Acquiring in-vehicle sound data of the target vehicle; If it is determined that there is noise according to the sound data, a target vehicle interior image is obtained, where the target vehicle interior image is an image of the vehicle interior at a location where a sound source corresponding to the noise is located; determining, based on the target vehicle interior image, the passenger type and current behavior of a target passenger, the target passenger being a passenger seated in a seat corresponding to the location of the sound source; If it is detected that the current behavior belongs to the preset reference interference behavior and the passenger type is a child, the first device is controlled to perform a first tangible reminder operation and play a first voice reminder until a closing reminder instruction sent by the second device is detected. The first tangible reminder operation includes a first vibration operation and / or a first ringing operation. The first voice reminder is used to remind the target passenger to sit down. The first device is a device worn by the target passenger, and the second device is a device held by the guardian of the target passenger.

2. The method according to claim 1, characterized in that The method further comprises: The vehicle system continuously acquires the target vehicle interior image to detect subsequent behavior of the target passenger; When the vehicle system detects that the subsequent behavior belongs to the reference interference behavior, it controls the first device to perform a second sensory reminder operation and play a second voice reminder, the second sensory reminder operation including a second vibration operation and / or a second ringing operation, and the second voice reminder is used to inquire about the video or cartoon that the target passenger wants to watch; and obtains the voice instruction of the target passenger; When the voice instruction indicates that the target passenger wants to watch the video or the cartoon, the vehicle system sends a play query instruction to the second device; The vehicle system plays the video or the cartoon on the rear display screen corresponding to the target passenger in response to the consent to play instruction sent by the second device in response to the play query instruction.

3. The method according to claim 2, characterized in that The method further comprises: When the vehicle system receives the stop playback instruction sent by the second device, the video playback record is saved according to the video playback progress, and the video playback record includes the video storage location, playback progress, and video name; If the playback progress is less than the preset progress, the vehicle system updates the second voice reminder according to the video name, and the updated second voice reminder is used to remind the target passenger whether to watch the video or the cartoon corresponding to the video name; When the vehicle system receives the voice instruction containing the video name, it continues to play the video or the cartoon according to the video playback record.

4. The method according to claim 1, wherein The target vehicle includes an on-board camera and a sound collection device, and the in-vehicle sound data includes sound sub-data collected by each of the sound collection devices; The step of acquiring the target vehicle interior image includes: The sound collection device performs a sound recognition operation on each of the sound sub-data in sequence to determine the target sound sub-data whose sound type is the noise; The vehicle system performs a sound source localization operation through a preset microphone according to the target sound sub-data to determine the location of the sound source; The vehicle system controls at least one of the vehicle-mounted cameras corresponding to the location of the sound source to capture a preset number of images inside the target vehicle.

5. The method according to claim 4, characterized in that The sound collection device sequentially performs a sound recognition operation on each of the sound sub-data to determine target sound sub-data whose sound type is the noise, including: The vehicle system determines the sound intensity, sound frequency and duration corresponding to each sound sub-data; If the sound intensity of the sound sub-data being processed is greater than a preset decibel, the vehicle system determines that the sound sub-data being processed is sound data to be identified; If the sound intensity of the sound sub-data being processed is less than or equal to a preset decibel, the vehicle system determines a target frequency level based on the sound frequency and a set of preset frequency ranges, where each preset frequency range in the set of preset frequency ranges corresponds to a frequency level; and if the target frequency level is greater than the preset frequency level and the corresponding duration is greater than a preset number of seconds, the sound sub-data being processed is determined to be the sound data to be identified; The vehicle system performs a speech recognition operation on the sound data to be identified to obtain a recognition text; If the recognized text represents a complete intent, the vehicle system determines that the sound type of the sound sub-data being processed is a conversational sound, and continues to process the next sound sub-data until the target sound sub-data is determined; If the recognized text representation is an incomplete intent, the vehicle system determines that the sound sub-data being processed is the target sound sub-data.

6. The method according to claim 2, characterized in that After determining the passenger type and current behavior of the target passenger based on the target vehicle interior image, the method further includes: If the vehicle system detects that the current behavior is abnormal, it controls the first device to perform a collection operation to obtain the current vital sign information of the target passenger, where the abnormal behavior is a behavior associated with the condition to be monitored that is pre-set by the guardian through the second device; The vehicle system obtains reference vital sign information of the target passenger, where the reference vital sign information is vital sign information collected by the first device when the target passenger is in a normal state; The vehicle system determines a comparison result based on the reference vital sign information and the current vital sign information; If the comparison result indicates that the current vital sign information is in an abnormal state, the vehicle system controls the second device to perform a third tangible reminder operation and play a third voice reminder until a confirmation instruction sent by the second device is received. The third tangible reminder operation is a third vibration operation, and the third voice reminder is used to remind the guardian to pay attention to the physical condition of the target passenger.

7. The method according to claim 6, characterized in that The vital sign information includes a plurality of basic vital sign information, each of which corresponds to a vital sign type; The vehicle system determines a comparison result based on the reference vital sign information and the current vital sign information, including: The vehicle system determines a plurality of abnormal physical sign information associated with the condition to be monitored, wherein the abnormal physical sign information is abnormal basic physical sign information of the patient when the condition to be monitored occurs; The vehicle system determines the vital sign change amount corresponding to each piece of abnormal vital sign information according to the reference vital sign information and the current vital sign information; The vehicle system obtains the normal fluctuation range corresponding to each abnormal physical sign information; If each of the vital sign changes is within the corresponding normal fluctuation range, the vehicle system determines that the current vital sign information is in a normal state and generates the comparison result; If the amount of change in the vital sign exceeds the corresponding normal fluctuation range, the vehicle system determines that the current vital sign information is in the abnormal state and generates the comparison result.

8. An intelligent processing device for rear-seat user behavior information in a cabin, characterized by: The vehicle-mounted system used in a target vehicle comprises: A first acquiring unit, configured to acquire in-vehicle sound data of the target vehicle; a second acquiring unit, configured to acquire a target vehicle interior image if it is determined based on the sound data that noise exists, the target vehicle interior image being an image of the vehicle interior at a location where a sound source corresponding to the noise is located; a determination unit, configured to determine a passenger type and current behavior of a target passenger based on the target vehicle interior image, wherein the target passenger is a passenger seated in a seat corresponding to the location of the sound source; A control unit is configured to control a first device to perform a first sensible reminder operation and play a first voice reminder if it is detected that the current behavior belongs to a preset reference interference behavior and the passenger type is a child, until a reminder closing instruction sent by a second device is detected, wherein the first sensible reminder operation includes a first vibration operation and / or a first ringing operation, and the first voice reminder is used to remind the target passenger to sit down. The first device is a device worn by the target passenger, and the second device is a device held by the guardian of the target passenger.

9. A vehicle computer system, characterized in that: The system comprises a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that A computer program for electronic data exchange is stored, wherein the computer program causes a computer to execute the method according to any one of claims 1 to 7.

Images