Data processing method and device and vehicle

Abstract

The embodiment of the invention discloses a data processing method and device and a vehicle. Video stream data collected by multiple camera modules are obtained, detection information is obtained based on the video stream data collected by one or multiple camera modules, the detection information comprises at least one of passenger taking information, passenger behavior information and in-vehicle environment information, and a processing scheme is generated according to the detection information and executed. Therefore, the detection information is acquired according to the video stream data acquired by the multi-path camera module, so that the shooting view angle is more comprehensive, and the refinement degree and reliability of the detection information are improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a data processing method, device and vehicle. Background Art

[0002] With the rapid development of autonomous driving technology, autonomous driving ride-hailing has become an important development direction for future transportation, and its safety and passenger experience have become the focus of industry attention. In-vehicle inspection is a key link in ensuring passenger safety, optimizing vehicle dispatching and improving service quality, and its technical implementation method is particularly important.

[0003] However, existing technologies for in-vehicle detection of self-driving online ride-hailing vehicles mostly use a single camera or radar solution. This single sensor detection method has obvious limitations and cannot achieve comprehensive coverage of the in-vehicle environment. Specifically, a single camera is limited by its viewing angle and installation position, and often has detection blind spots, unable to capture information from all key areas in the car, resulting in insufficient reliability of detection results. Although radar can detect the existence and location of objects, it has deficiencies in detail capture and image recognition, making it difficult to meet the refined needs of in-vehicle detection. Summary of the invention

[0004] In view of this, an object of an embodiment of the present invention is to provide a data processing method, device and vehicle, which can improve the refinement and reliability of detection information.

[0005] In a first aspect, an embodiment of the present invention provides a data processing method, the method comprising:

[0006] Obtain video stream data collected by multiple camera modules;

[0007] Acquiring detection information based on video stream data collected by one or more camera modules, wherein the detection information includes at least one of passenger riding information, passenger behavior information, and in-vehicle environment information;

[0008] A processing plan is generated and executed according to the detection information.

[0009] In some embodiments, obtaining video stream data collected by multiple camera modules includes:

[0010] Acquire first video stream data collected by a first camera module, where the first camera module is used to collect data of a front row of the vehicle and data of part of a rear row of the vehicle;

[0011] Acquire second video stream data collected by a second camera module, where the second camera module is used to collect data from the rear row of the vehicle;

[0012] A third video stream data collected by a third camera module is obtained, where the third camera module is used to collect data from a trunk of a vehicle.

[0013] In some embodiments, the passenger riding information includes at least one of the number of passengers, the type of passengers, and the wearing status of seat belts; the number of passengers includes the number of people in the back row of the vehicle and the number of people in the trunk; the passenger behavior information includes at least one of dangerous goods, passenger position, multi-person limb behavior, single-person limb movement, and breathing rate; the in-vehicle environment information includes at least one of irregular behavior, in-vehicle hygiene, pets, and lost items.

[0014] In some embodiments, the acquiring detection information based on the video stream data collected by one or more camera modules includes:

[0015] The passenger position is obtained based on the first video stream data.

[0016] In some embodiments, the acquiring detection information based on the video stream data collected by one or more camera modules includes:

[0017] Based on the second video stream data, at least one of the number of people in the rear seat of the vehicle, the type of passengers, the wearing of seat belts, the limb movements of multiple people, the limb movements of a single person, the breathing frequency and irregular behaviors is obtained.

[0018] In some embodiments, the acquiring detection information based on the video stream data collected by one or more camera modules includes:

[0019] The trunk occupancy status is obtained based on the third video stream data.

[0020] In some embodiments, the acquiring detection information based on the video stream data collected by one or more camera modules includes:

[0021] At least one of the number of passengers, dangerous goods, hygiene in the vehicle, pets, and lost items is obtained based on the second video stream data and the third video stream data.

[0022] In some embodiments, the acquiring detection information based on the video stream data collected by one or more camera modules includes:

[0023] Obtain occlusion conditions based on the first video stream data, the second video stream data, and the third video stream data.

[0024] In some embodiments, generating a processing plan based on the detection information and executing the processing plan includes:

[0025] The processing scheme is determined and executed according to the comparison result between the detection information and the predetermined threshold.

[0026] In some embodiments, the processing scheme includes sending a prompt message through the vehicle system and / or uploading the prompt message to a server.

[0027] In some embodiments, the method further comprises:

[0028] Obtaining audio stream data collected by at least one audio collection module;

[0029] An audio and video service is performed based on the audio stream data and the video stream data.

[0030] In some embodiments, the audio and video service includes at least one of live intercom, video recording, video storage, video extraction, video locking, video encryption, image extraction, and watermarking.

[0031] In some embodiments, the method further comprises:

[0032] Execution algorithm management;

[0033] Among them, the algorithm management includes at least one of algorithm registration, algorithm lifecycle management, algorithm scheduling, configuration of the algorithm rule engine and the container for the algorithm model to run, and algorithm full-link detection.

[0034] In some embodiments, the method further comprises:

[0035] Perform device-based services;

[0036] Among them, the device basic services include at least one of device registration and authentication, command configuration issuance, log service, basic data, multimedia components, over-the-air download technology, operation control and system recovery.

[0037] In a second aspect, an embodiment of the present invention provides a vehicle, the vehicle comprising:

[0038] Multi-channel camera module, used for collecting video stream data;

[0039] A memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the following steps:

[0040] Obtain video stream data collected by multiple camera modules;

[0041] Acquiring detection information based on video stream data collected by one or more camera modules, wherein the detection information includes at least one of passenger riding information, passenger behavior information, and in-vehicle environment information;

[0042] A processing plan is generated and executed according to the detection information.

[0043] In some embodiments, the vehicle further comprises:

[0044] At least one audio acquisition module for acquiring audio stream data;

[0045] The one or more computer program instructions are executed by the processor to implement the following steps:

[0046] An audio and video service is performed based on the audio stream data and the video stream data.

[0047] In some embodiments, the multi-channel camera module includes:

[0048] A first camera module, used for collecting first video stream data, wherein the first video stream data includes data of the front row of the vehicle and data of part of the rear row of the vehicle;

[0049] A second camera module, used for collecting second video stream data, wherein the second video stream data includes data of the rear row of the vehicle;

[0050] The third camera module is used to collect third video stream data, and the third video stream data includes data of the vehicle trunk.

[0051] In some embodiments, the vehicle further comprises:

[0052] The communication module is used to communicate with the server.

[0053] In some embodiments, the vehicle further comprises:

[0054] Car computer system;

[0055] The one or more computer program instructions are executed by the processor to implement the following steps:

[0056] Control the vehicle system to issue a prompt message.

[0057] In a third aspect, an embodiment of the present invention provides a data processing device, the device comprising:

[0058] A video stream data acquisition unit, used to acquire video stream data collected by multiple camera modules;

[0059] A detection information acquisition unit, configured to acquire detection information based on video stream data collected by one or more camera modules, wherein the detection information includes at least one of passenger riding information, passenger behavior information, and in-vehicle environment information;

[0060] A solution execution unit is used to generate and execute a processing solution based on the detection information.

[0061] In a fourth aspect, an embodiment of the present invention provides a computer program product, wherein the computer program product includes a computer program. When the computer program runs on a computer, the computer executes the method described in the first aspect.

[0062] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0063] The technical solution of the embodiment of the present invention obtains the video stream data collected by multiple camera modules, obtains the detection information based on the video stream data collected by one or more camera modules, and the detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information, and generates and executes the processing scheme according to the detection information. Therefore, by obtaining the detection information based on the video stream data collected by the multiple camera modules, the shooting angle is more comprehensive, and the refinement and reliability of the detection information are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0065] Figure 1 is a schematic diagram of a data processing system according to an embodiment of the present invention;

[0066] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention;

[0067] Figure 3 is a schematic diagram of the corresponding relationship between the detection information and the camera module according to an embodiment of the present invention;

[0068] Figure 4 Schematic diagram of a data processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0070] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0071] At the same time, it should be understood that in the following description, "circuit" refers to a conductive loop composed of at least one element or subcircuit through electrical connection or electromagnetic connection. When an element or circuit is said to be "connected to" another element or an element / circuit is said to be "connected between" two nodes, it can be directly coupled or connected to another element or there can be an intermediate element, and the connection between the elements can be physical, logical, or a combination thereof. On the contrary, when an element is said to be "directly coupled to" or "directly connected to" another element, it means that there is no intermediate element between the two.

[0072] Unless the context clearly requires otherwise, the words "include", "comprising" and similar words throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, the meaning is "including but not limited to".

[0073] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0074] Figure 1 Schematic diagram of a data processing system according to an embodiment of the present invention. Figure 1 As shown, the data processing system of the embodiment of the present invention includes a server 1 and a vehicle 2. The server 1 and the vehicle 2 are connected in communication.

[0075] Among them, server 1 is the management platform of vehicle 2, which is used to manage the vehicle, including obtaining relevant information of the vehicle and sending instructions to the vehicle.

[0076] Vehicle 2 is an autonomous driving online car-hailing vehicle used to provide online car-hailing services.

[0077] The vehicle 2 includes a processor 21 , a memory 22 , a multi-channel camera module, an audio acquisition module 26 , a vehicle system 27 and a communication module 28 .

[0078] The memory 22 is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor 21 to implement the data processing method of the embodiment of the present invention.

[0079] The multi-channel camera module is used to collect video stream data. Specifically, the multi-channel camera module includes a first camera module 23, a second camera module 24 and a third camera module 25.

[0080] Among them, the first camera module 23 is used to collect the first video stream data, and the first video stream data includes the data of the front row of the vehicle and part of the rear row of the vehicle. Specifically, the first camera module 23 is arranged in the front row of the vehicle and installed in the center, and the shooting direction is configured to capture the data of the front row of the vehicle and part of the rear row of the vehicle. Among them, the data of the front row of the vehicle includes the face, torso, thighs, etc. of the front row passengers, as well as the steering wheel, the front row seat cushions, backrests, headrests, wearing of front seat belts, etc. The data of part of the rear row of the vehicle includes the middle seat cushion, backrest, headrest, etc. of the rear row.

[0081] The second camera module 24 is used to collect the second video stream data, which includes the data of the rear row of the vehicle. Specifically, the second camera module is arranged in the rear row and installed in the center, and the shooting direction is configured to shoot the data of the rear row of the vehicle. Among them, the data of the rear row of the vehicle includes the face, torso, thigh, rear seat cushion, backrest, headrest, rear seat belt wearing, etc. of the rear passengers, and at the same time, as much as possible of the foot pad is covered, the door panel armrest can be seen, and the area where the water cup is placed on the door can be seen as much as possible.

[0082] The third camera module 25 is used to collect third video stream data, and the third video stream data includes data of the vehicle trunk. Specifically, the third camera module is arranged in the trunk, and the shooting direction is configured to cover the entire bottom plate of the trunk.

[0083] In some embodiments, the resolution of the multi-channel camera module is 1280*720, the frame rate is 25fps, the bit rate is greater than 500K, and the picture quality requires that when collecting video streams, the noise of the native image should not be too much, especially the noise on the face at night; at the same time, it can support night vision (switch to IR, can see through some sunglasses), and there must be a fill light; the camera data format is any one of the three NV21, YUY2, and ARGB.

[0084] In some embodiments, the data storage medium of the embodiment of the present invention is a storage space allocated inside the vehicle computer, with a size of 32G. Among them, the road test data is the data of the past 4 hours, and the operation is theoretically the data within 72 hours. Considering that there is no need to store when there is no one in the car, the operation is the data of the past 52 hours of the in-vehicle camera module. For the first video stream data collected by the first camera module, the storage time is 24h, the storage size is 7G, ​​the vehicle side is compressed to a size of 640*480, and the original route is uploaded. For the second video stream data collected by the second camera module, the storage time is 50h, the storage size is 14.6G, the vehicle side is compressed to a size of 640*480, and the original route is uploaded. For the third video stream data collected by the third camera module, the storage time is 24h, the storage size is 7G, ​​the vehicle side is compressed to a size of 640*480, and the original route is uploaded. Among them, the original route upload refers to the process of capturing the image data inside the vehicle through the camera module, and transmitting these data to the processor in real time for processing and analysis.

[0085] In some embodiments, the processor is an existing processor of the vehicle computer, thereby making full use of the video processing capability and storage capability of the vehicle computer, greatly reducing the cost of the independent host. At the same time, the computing power of the processor is 30k DMIPS (Dhrystone Million Instructions executed Per Second, used to measure integer computing power). The computing power of the processor can be configured according to actual needs.

[0086] In some embodiments, the vehicle system uses the Android system and can be directly adapted.

[0087] In some embodiments, the original video stream data 1080P of the camera module supports 3-channel OMS (Occupancy Monitoring System). Each OMS simultaneously outputs YVU stream (algorithm), recording stream, live stream, or outputs any 2 and 1 of the YVU stream, recording stream, and live stream, which can be flexibly turned on / off according to business requirements.

[0088] Among them, the video stream resolution and frame rate of the three camera modules after compression by the audio and video SDK (Software Development Kit) are as follows:

[0089] YVU stream: The image is 720P and no less than 10 frames.

[0090] Recording stream: minimum 480P, no less than 25 frames.

[0091] Live streaming: 480P, no less than 15 frames.

[0092] Therefore, by reasonably setting the positions and shooting directions of the first camera module 23, the second camera module 24 and the third camera module 25, the environment inside the vehicle can be covered as comprehensively as possible, thereby improving the refinement and reliability of the detection information.

[0093] The processor 21 obtains video stream data collected by multiple camera modules, obtains detection information based on the video stream data collected by one or more camera modules, and generates and executes a processing plan based on the detection information.

[0094] The audio collection module 26 is a microphone, which is used to collect audio stream data. The audio collection module 26 can be one channel or multiple channels.

[0095] The processor 21 obtains audio stream data collected by at least one audio collection module, and performs audio and video services based on the audio stream data and the video stream data.

[0096] The vehicle system 27 is a human-machine interaction system inside the vehicle, including a front vehicle system and a rear vehicle system. The vehicle system includes a processing module and a display screen.

[0097] The processing solution generated by the processor 21 includes sending a prompt message through the vehicle system.

[0098] The communication module 28 is used to communicate with the server 1 .

[0099] The processor 21 is used to communicate with the server through the communication module 28, including uploading prompt information to the server, uploading detection information to the server, uploading audio and video information to the server, and receiving control instructions sent by the server.

[0100] It should be noted that Figure 1 Only some modules and circuits of the vehicle are shown. Figure 1 In addition to what is shown in the figure, the vehicle also includes modules such as speakers for playing sounds.

[0101] The embodiment of the present invention obtains the video stream data collected by multiple camera modules, obtains the detection information based on the video stream data collected by one or more camera modules, and the detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information, and generates and executes a processing scheme according to the detection information. Therefore, by obtaining the detection information based on the video stream data collected by the multiple camera modules, the shooting angle is more comprehensive, and the refinement and reliability of the detection information are improved.

[0102] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 2As shown, the data processing method of the embodiment of the present invention is executed by a processor in a vehicle, and specifically includes the following steps:

[0103] Step S100: Acquire video stream data collected by multiple camera modules.

[0104] In this embodiment, the first video stream data collected by the first camera module is obtained, and the first camera module is used to collect data of the front row of the vehicle and part of the rear row of the vehicle. The second video stream data collected by the second camera module is obtained, and the second camera module is used to collect data of the rear row of the vehicle. The third video stream data collected by the third camera module is obtained, and the third camera module is used to collect data of the trunk of the vehicle.

[0105] Step S200: Acquire detection information based on video stream data collected by one or more camera modules.

[0106] In this embodiment, based on the video stream data collected by one or more camera modules, image information is extracted from the video stream data, and the image information is recognized by a predetermined image recognition algorithm to obtain detection information.

[0107] The detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information. The passenger riding information includes at least one of the number of passengers, passenger type and seat belt wearing status; the number of passengers includes the number of passengers in the rear seats and the number of passengers in the trunk; the passenger behavior information includes at least one of dangerous goods, passenger position, multi-person limb behavior, single-person limb movement and breathing rate; the in-vehicle environment information includes at least one of irregular behavior, in-vehicle hygiene, pets and lost items.

[0108] Specifically, Figure 3 FIG. 1 is a schematic diagram of the corresponding relationship between the detection information and the camera module according to an embodiment of the present invention. Figure 3 , based on the video stream data collected by one or more camera modules, the detection information is obtained including:

[0109] Step S210: Obtain the passenger position based on the first video stream data.

[0110] In this embodiment, the first video stream data collected by the first camera module is obtained, a first image is obtained according to the first video stream data, and the first image is recognized based on a predetermined image recognition algorithm to obtain the passenger position. The first image is a video frame extracted from the first video stream data.

[0111] As described above, the first camera module is arranged in the front row of the vehicle and installed in the center, and the shooting direction is configured to capture data of the front row of the vehicle and part of the rear row of the vehicle. Among them, the data of the front row of the vehicle includes the face, torso, thighs, etc. of the front row passengers, as well as the steering wheel, the front row seat cushions, backrests, headrests, wearing of front seat belts, etc. The data of the rear row of the vehicle includes the middle seat cushion, backrest, headrest, etc. in the rear row. Thus, the passenger position can be detected by identifying the video frames in the first video stream data captured by the first camera module. The passenger position is whether part of the passenger's body enters the front row area. Thus, it is possible to detect abnormal behaviors such as passengers damaging items in the car or attempting to touch the steering wheel.

[0112] Step S220: obtaining at least one of the number of people in the rear seat of the vehicle, the type of passengers, the wearing of seat belts, the limb movements of multiple people, the limb movements of a single person, the breathing frequency and irregular behaviors based on the second video stream data.

[0113] In this embodiment, the second video stream data collected by the second camera module is obtained, a second image is obtained according to the second video stream data, and the second image is recognized based on a predetermined image recognition algorithm to obtain at least one of the number of people in the rear row of the vehicle, the type of passengers, the wearing of seat belts, the limb behavior of multiple people, the limb movement of a single person, the breathing frequency, and irregular behavior. The second image is a video frame extracted from the second video stream data.

[0114] Among them, for the number of people in the back row of the vehicle and the specific seats (rear row left, middle and right position identification), it can detect whether there is someone in the car, including the back row and the trunk.

[0115] Passenger type Whether a single passenger at a specific location is a minor can be determined by detecting the passenger's age.

[0116] Seat belt wearing condition, whether a single passenger in a specific position has fastened his seat belt correctly, can detect whether the seat belt is worn correctly.

[0117] Multi-person physical behavior refers to physical contact recognition, such as physical contact. It can detect physical conflicts between passengers in the car, etc.

[0118] Single-person body movements include lying down, bending over, holding the chest, and struggling (large movements of the limbs), which can be used to detect abnormalities in the passenger himself.

[0119] Respiratory rate refers to the recognition of respiratory rate, which can detect abnormalities in the passengers themselves.

[0120] Irregular behaviors are detected through smoking model behavior, including strip objects, sparks, and smoke recognition, in order to identify passengers' irregular behaviors, such as smoking, drunkenness, and vomiting.

[0121] Step S230: Obtain the trunk passenger status based on the third video stream data.

[0122] In this embodiment, the third video stream data collected by the third camera module is obtained, a third image is obtained according to the third video stream data, and the third image is recognized based on a predetermined image recognition algorithm to obtain the trunk occupancy status. The third image is a video frame extracted from the third video stream data.

[0123] The trunk occupancy status includes whether there is someone in the trunk or not, which can be identified by the human body part in the field of view.

[0124] Step S240: obtaining at least one of the number of passengers, dangerous goods, in-vehicle hygiene, pets, and lost items based on the second video stream data and the third video stream data.

[0125] In this embodiment, second video stream data captured by the second camera module is obtained, a second image is obtained based on the second video stream data, third video stream data captured by the third camera module is obtained, a third image is obtained based on the third video stream data, and based on the combination of the second image and the third image, a predetermined image recognition algorithm is used to obtain at least one of the number of passengers, dangerous goods, vehicle hygiene, pets, and lost items.

[0126] The number of passengers includes the number of people in the back row identified by the second image and the number of people in the trunk identified by the third image, so as to realize overload detection.

[0127] Dangerous items include controlled knives, fireworks and firecrackers, etc. It can detect whether passengers are carrying prohibited items.

[0128] The in-car hygiene includes the identification of paper balls, water stains, and stains on the ground and rear seats to detect the hygiene conditions in the car.

[0129] Pet identification includes identification of cats, dogs and other pets to achieve sanitary pet identification detection in the car.

[0130] Lost items is the identification of commonly lost items, such as mobile phones, backpacks, suitcases, wallets, water cups, etc.

[0131] Step S250: Obtain occlusion conditions based on the first video stream data, the second video stream data, and the third video stream data.

[0132] In this embodiment, the first video stream data collected by the first camera module is obtained, and the first image is obtained according to the first video stream data. The second video stream data collected by the second camera module is obtained, and the second image is obtained according to the second video stream data. The third video stream data collected by the third camera module is obtained, and the third image is obtained according to the third video stream data. Based on the first image, the second image, and the third image, the occlusion situation is stupidly identified using a predetermined image recognition algorithm. The occlusion situation identification can be used to detect device abnormalities, including the camera being covered or having stains that affect the camera detection.

[0133] It should be noted that the various detection information in the above steps S210-S250 are all realized by an image recognition algorithm. The recognition of different detection information can be realized by one image recognition algorithm or based on different image recognition algorithms. That is, the image detection algorithm of the embodiment of the present invention can realize the recognition of one or more detection information.

[0134] In some embodiments, the image recognition algorithm of the embodiment of the present invention supports nighttime recognition. Correspondingly, the first camera module, the second camera module and the third camera module all have night vision functions.

[0135] In some embodiments, the embodiments of the present invention are based on OMS (Occupancy Monitoring System), and obtain video streams through camera modules to obtain image data, which are then handed over to NPU (Neural-network Processing Unit) for further analysis and learning to obtain detection information. Among them, the image recognition algorithm of the embodiments of the present invention is implemented based on NPU.

[0136] Step S300: Generate and execute a processing plan based on the detection information.

[0137] In this embodiment, the processing scheme is determined and executed according to the comparison result between the detection information and the predetermined threshold.

[0138] Specifically, after obtaining each piece of detection information, each piece of detection information is compared with a predetermined threshold to obtain a comparison result, and then abnormal detection information is obtained. For the abnormal detection information, a corresponding processing solution is determined and executed.

[0139] Among them, the processing plan includes sending prompt information through the vehicle system, uploading prompt information to the server, making an emergency call, etc.

[0140] For example, if it is detected that a passenger is not wearing a seat belt correctly, a prompt message will be sent through the vehicle system to remind the passenger to wear the seat belt correctly.

[0141] For another example, if it is detected that the passenger's breathing rate is abnormal, an emergency call is made or a prompt message is uploaded to the server so that the staff can manually process it based on the uploaded prompt message.

[0142] For another example, if there is a problem with the hygiene in the car, a prompt message will be uploaded to the server so that the staff can clean it.

[0143] It should be noted that the embodiments of the present invention have different corresponding processing solutions for different abnormal situations, which can be configured according to actual needs.

[0144] The embodiment of the present invention obtains the video stream data collected by multiple camera modules, obtains the detection information based on the video stream data collected by one or more camera modules, and the detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information, and generates and executes a processing scheme according to the detection information. Therefore, by obtaining the detection information based on the video stream data collected by the multiple camera modules, the shooting angle is more comprehensive, and the refinement and reliability of the detection information are improved.

[0145] In some embodiments, the data processing method further includes:

[0146] Step S400: Acquire audio stream data collected by at least one audio collection module.

[0147] In this embodiment, the audio collection module may be one channel or multiple channels.

[0148] Step S500: Execute audio and video services based on the audio stream data and the video stream data.

[0149] In this embodiment, video stream data collected by multiple camera modules and audio stream data collected by at least one audio collection module are obtained, and audio and video services are performed based on the audio stream data and the video stream data. The audio and video services include at least one of live intercom, video recording, video storage, video extraction, video locking, video encryption, image extraction, and watermarking.

[0150] For live intercom, you can initiate / close live preview, make live call intercom, etc.

[0151] For video recording, basic video recording and storage, flameout delay recording, flameout wake-up recording, etc. can be performed.

[0152] For video storage, it can support video storage path configuration, video storage duration configuration, video cleanup configuration, etc.

[0153] For video extraction, the video can be extracted to the server. Since the server has more sufficient storage space, the video storage time can be extended.

[0154] For video lock, emergency video lock is not covered (depending on collision algorithm or airbag deployment). Video of abnormal situation can be retained for a longer time.

[0155] For video encryption, you can configure video encryption (locally exported files cannot be viewed directly and need to be decrypted), which can improve data security.

[0156] For image extraction, you can extract pictures (extract frames) from historical videos and take photos in real time, and you can also upload pictures within minutes.

[0157] For adding watermarks, you can add timestamps, tags and other information to the video, and you can also provide a time watermark.

[0158] Therefore, the audio and video inside the car can be recorded through the audio and video service, further improving the reliability of in-car detection.

[0159] In some embodiments, the data processing method further includes:

[0160] Step S600: Execute algorithm management.

[0161] In this embodiment, the algorithm management includes at least one of algorithm registration, algorithm lifecycle management, algorithm scheduling, configuration of an algorithm rule engine and a container for algorithm model operation, and algorithm full-link detection.

[0162] Therefore, managing the algorithm is beneficial to the iterative update of the vehicle.

[0163] In some embodiments, the data processing method further includes:

[0164] Step S700: Execute device basic services.

[0165] In this embodiment, the device basic service includes at least one of device registration and authentication, command configuration issuance, log service, basic data, multimedia components, over-the-air download technology, operation control and system recovery.

[0166] Device registration and authentication is the device networking registration and authentication.

[0167] The command configuration sent includes audio and video parameter configuration, algorithm parameter configuration, etc.

[0168] Log services include audio and video logs, algorithm logs, system logs, etc.

[0169] Basic data includes positioning information and sensor information.

[0170] Multimedia components include video cropping, video frame extraction, video compression, etc.

[0171] Over-the-air download technology can realize the linkage upgrade of multiple applications.

[0172] Operation control includes restarting services (restarting the system), command transmission (SDK receives commands and forwards them to other applications), etc.

[0173] System recovery is to achieve rapid recovery and normal operation of business through pre-established recovery plans and technical means in the face of emergencies such as system failures, hardware failures, network failures, etc.

[0174] Therefore, the normal operation of the vehicle can be guaranteed through equipment basic services.

[0175] It should be noted that the embodiments of the present invention are described using an autonomous driving online-hailing car as an example, but the present invention does not limit the usage scenarios of the data processing method, and it can be applied to various scenarios, such as autonomous driving vehicles in other scenarios, non-autonomous driving online-hailing cars or taxis, and various other types of vehicles.

[0176] The embodiment of the present invention obtains the video stream data collected by multiple camera modules, obtains the detection information based on the video stream data collected by one or more camera modules, and the detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information, and generates and executes a processing scheme according to the detection information. Therefore, by obtaining the detection information based on the video stream data collected by the multiple camera modules, the shooting angle is more comprehensive, and the refinement and reliability of the detection information are improved.

[0177] Figure 4 Schematic diagram of a data processing device according to an embodiment of the present invention. Figure 4 As shown, the data processing device of the embodiment of the present invention includes the device including a video stream data acquisition unit 41, a detection information acquisition unit 42 and a solution execution unit 43. The video stream data acquisition unit 41 is used to acquire video stream data collected by multiple camera modules. The detection information acquisition unit 42 is used to acquire detection information based on the video stream data collected by one or more camera modules, and the detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information. The solution execution unit 43 is used to generate a processing solution according to the detection information and execute it.

[0178] In some embodiments, the video stream data acquisition unit includes:

[0179] A first acquisition subunit, used to acquire first video stream data acquired by a first camera module, wherein the first camera module is used to acquire data of a front row of the vehicle and data of part of a rear row of the vehicle;

[0180] A second acquisition subunit is used to acquire second video stream data acquired by a second camera module, where the second camera module is used to acquire data of the rear row of the vehicle;

[0181] The third acquisition subunit is used to acquire the third video stream data collected by the third camera module, and the third camera module is used to collect data from the trunk of the vehicle.

[0182] In some embodiments, the passenger riding information includes at least one of the number of passengers, the type of passengers, and the wearing status of seat belts; the number of passengers includes the number of people in the back row of the vehicle and the number of people in the trunk; the passenger behavior information includes at least one of dangerous goods, passenger position, multi-person limb behavior, single-person limb movement, and breathing rate; the in-vehicle environment information includes at least one of irregular behavior, in-vehicle hygiene, pets, and lost items.

[0183] In some embodiments, the acquiring detection information based on the video stream data collected by one or more camera modules includes:

[0184] The passenger position is obtained based on the first video stream data.

[0185] In some embodiments, the detection information acquisition unit is used to:

[0186] Based on the second video stream data, at least one of the number of people in the rear seat of the vehicle, the type of passengers, the wearing of seat belts, the limb movements of multiple people, the limb movements of a single person, the breathing frequency and irregular behaviors is obtained.

[0187] In some embodiments, the detection information acquisition unit is used to:

[0188] The trunk occupancy status is obtained based on the third video stream data.

[0189] In some embodiments, the detection information acquisition unit is used to:

[0190] At least one of the number of passengers, dangerous goods, hygiene in the vehicle, pets, and lost items is obtained based on the second video stream data and the third video stream data.

[0191] In some embodiments, the detection information acquisition unit is used to:

[0192] Obtain occlusion conditions based on the first video stream data, the second video stream data, and the third video stream data.

[0193] In some embodiments, the scheme execution unit is used to:

[0194] The processing scheme is determined and executed according to the comparison result between the detection information and the predetermined threshold.

[0195] In some embodiments, the processing scheme includes sending a prompt message through the vehicle system and / or uploading the prompt message to a server.

[0196] In some embodiments, the apparatus further comprises:

[0197] An audio stream data acquisition unit, used to acquire audio stream data collected by at least one audio collection module;

[0198] An audio and video service unit is used to perform audio and video services based on the audio stream data and the video stream data.

[0199] In some embodiments, the audio and video service includes at least one of live intercom, video recording, video storage, video extraction, video locking, video encryption, image extraction, and watermarking.

[0200] In some embodiments, the apparatus further comprises:

[0201] An algorithm management unit, used for performing algorithm management;

[0202] Among them, the algorithm management includes at least one of algorithm registration, algorithm lifecycle management, algorithm scheduling, configuration of the algorithm rule engine and the container for the algorithm model to run, and algorithm full-link detection.

[0203] In some embodiments, the apparatus further comprises:

[0204] Basic service unit, used to perform basic device services;

[0205] Among them, the device basic services include at least one of device registration and authentication, command configuration issuance, log service, basic data, multimedia components, over-the-air download technology, operation control and system recovery.

[0206] The embodiment of the present invention obtains the video stream data collected by multiple camera modules, obtains the detection information based on the video stream data collected by one or more camera modules, and the detection information includes at least one of passenger riding information, passenger behavior information and in-vehicle environment information, and generates and executes a processing scheme according to the detection information. Therefore, by obtaining the detection information based on the video stream data collected by the multiple camera modules, the shooting angle is more comprehensive, and the refinement and reliability of the detection information are improved.

[0207] exist Figure 1In the vehicle shown, the processor 21 and the memory 22 are connected via a bus. The memory 22 is suitable for storing instructions or programs executable by the processor 21. The processor 21 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 21 executes the instructions stored in the memory 22 to execute the method flow of the embodiment of the present invention as described above to process data and control other devices. The bus connects the above-mentioned multiple components together.

[0208] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices (equipment) or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0209] The present invention is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0210] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0211] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0212] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0213] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: Obtain video stream data collected by multiple camera modules; Acquiring detection information based on video stream data collected by one or more camera modules, wherein the detection information includes at least one of passenger riding information, passenger behavior information, and in-vehicle environment information; A processing plan is generated and executed according to the detection information.

2. The method according to claim 1, characterized in that The method of obtaining video stream data collected by multiple camera modules includes: Acquire first video stream data collected by a first camera module, where the first camera module is used to collect data of a front row of the vehicle and data of part of a rear row of the vehicle; Acquire second video stream data collected by a second camera module, where the second camera module is used to collect data from the rear row of the vehicle; A third video stream data collected by a third camera module is obtained, where the third camera module is used to collect data from a trunk of a vehicle.

3. The method according to claim 2, characterized in that The passenger riding information includes at least one of the number of passengers, passenger type and seat belt wearing status; the number of passengers includes the number of people in the back row of the vehicle and the number of people in the trunk; the passenger behavior information includes at least one of dangerous goods, passenger position, multi-person limb behavior, single-person limb movement and breathing rate; the in-vehicle environment information includes at least one of irregular behavior, in-vehicle hygiene, pets and lost items.

4. The method according to claim 3, characterized in that The acquisition of detection information based on the video stream data collected by one or more camera modules includes: The passenger position is obtained based on the first video stream data.

5. The method according to claim 3, characterized in that: The acquisition of detection information based on the video stream data collected by one or more camera modules includes: Based on the second video stream data, at least one of the number of people in the rear seat of the vehicle, the type of passengers, the wearing of seat belts, the limb movements of multiple people, the limb movements of a single person, the breathing frequency and irregular behaviors is obtained.

6. The method according to claim 3, characterized in that: The acquisition of detection information based on the video stream data collected by one or more camera modules includes: The trunk occupancy status is obtained based on the third video stream data.

7. The method according to claim 3, characterized in that The acquisition of detection information based on the video stream data collected by one or more camera modules includes: At least one of the number of passengers, dangerous goods, hygiene in the vehicle, pets, and lost items is obtained based on the second video stream data and the third video stream data.

8. The method according to claim 3, characterized in that The acquisition of detection information based on the video stream data collected by one or more camera modules includes: Obtain occlusion conditions based on the first video stream data, the second video stream data, and the third video stream data.

9. The method according to claim 1, characterized in that: Generating a processing plan based on the detection information and executing the plan includes: The processing scheme is determined and executed according to the comparison result between the detection information and the predetermined threshold.

10. The method according to claim 1, characterized in that The processing scheme includes sending a prompt message through the vehicle system and / or uploading the prompt message to the server.

11. The method according to claim 1, characterized in that: The method further comprises: Obtaining audio stream data collected by at least one audio collection module; An audio and video service is performed based on the audio stream data and the video stream data.

12. The method according to claim 11, characterized in that The audio and video service includes at least one of live intercom, video recording, video storage, video extraction, video locking, video encryption, image extraction, and watermarking.

13. The method according to claim 1, characterized in that The method further comprises: Execution algorithm management; Among them, the algorithm management includes at least one of algorithm registration, algorithm lifecycle management, algorithm scheduling, configuration of the algorithm rule engine and the container for the algorithm model to run, and algorithm full-link detection.

14. The method according to claim 1, characterized in that The method further comprises: Perform device-based services; The basic device services include at least one of device registration and authentication, command configuration delivery, log service, basic data, multimedia components, over-the-air download technology, operation control and system recovery.

15. A vehicle, characterized in that: The vehicle comprises: Multi-channel camera module, used for collecting video stream data; A memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the following steps: Obtain video stream data collected by multiple camera modules; Acquiring detection information based on video stream data collected by one or more camera modules, wherein the detection information includes at least one of passenger riding information, passenger behavior information, and in-vehicle environment information; A processing plan is generated and executed according to the detection information.

16. The vehicle according to claim 15, characterized in that The vehicle further comprises: At least one audio acquisition module for acquiring audio stream data; The one or more computer program instructions are executed by the processor to implement the following steps: An audio and video service is performed based on the audio stream data and the video stream data.

17. The vehicle according to claim 15, characterized in that The multi-channel camera module comprises: A first camera module, used for collecting first video stream data, wherein the first video stream data includes data of the front row of the vehicle and data of part of the rear row of the vehicle; A second camera module, used for collecting second video stream data, wherein the second video stream data includes data of the rear row of the vehicle; The third camera module is used to collect third video stream data, and the third video stream data includes data of the vehicle trunk.

18. The vehicle according to claim 15, characterized in that The vehicle further comprises: The communication module is used to communicate with the server.

19. The vehicle according to claim 15, characterized in that The vehicle further comprises: Car computer system; The one or more computer program instructions are executed by the processor to implement the following steps: Control the vehicle system to issue a prompt message.

20. A data processing device, characterized in that: The device comprises: A video stream data acquisition unit, used to acquire video stream data collected by multiple camera modules; A detection information acquisition unit, configured to acquire detection information based on video stream data collected by one or more camera modules, wherein the detection information includes at least one of passenger riding information, passenger behavior information, and in-vehicle environment information; A solution execution unit is used to generate and execute a processing solution based on the detection information.

21. A computer program product, comprising a computer program, characterized in that: When the computer program is executed on a computer, the computer executes the method according to any one of claims 1 to 14.

22. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: The computer program instructions, when executed by a processor, implement the method of any one of claims 1-14.

Images