Vehicle remote driving control method and system and electronic equipment

Abstract

The invention relates to a vehicle remote driving control method and system and electronic equipment. The method comprises the following steps: after establishing communication connection with a cloud platform, receiving video stream data uploaded by a vehicle end in real time from the cloud platform; the video stream data comprises video data which are acquired by the vehicle end and correspond to at least four different directions of the vehicle end; identifying video data of at least four different orientations in the video stream data, synchronously rendering the video data of at least four different orientations in a preset video window, displaying camera pictures of at least four different orientations of the vehicle end, and detecting a driving starting instruction; and after determining that the remote driving starting instruction is received, controlling the remote driving end to enter a remote control mode. According to the scheme provided by the invention, the real-time visualization of the surrounding environment of the vehicle can be realized, so that a remote operator can synchronously obtain the multi-dimensional environment information of the vehicle, and the real-time and comprehensive requirements of remote cab control in the automatic driving process are met.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and particularly to a method, system, and electronic device for remote driving control of a vehicle. Background Art

[0002] In the field of autonomous driving, video push-pull streaming technology is a key component for realizing real-time environment perception, remote collaboration and monitoring, data sharing and analysis.

[0003] In related technologies, the video stream interaction process during autonomous driving is mainly based on a single-process architecture, usually only supporting pushing one or two video streams, which are used to collect driver operation data and vehicle road data, resulting in being unable to be applied to application scenarios with real-time and comprehensive requirements for vehicle surrounding environment data. For example, it cannot meet the real-time and comprehensive requirements of remote cockpit control during autonomous driving, restricting the technological development of remote cockpit control during autonomous driving. Summary of the Invention

[0004] To solve or partially solve the problems existing in related technologies, this application provides a method, system, and electronic device for remote driving control of a vehicle, which can realize real-time visualization of the vehicle's surrounding environment, enabling remote operators to synchronously obtain multi-dimensional environment information of the vehicle and meeting the real-time and comprehensive requirements of remote cockpit control during autonomous driving.

[0005] The first aspect of this application provides a method for remote driving control of a vehicle, which is applied to a remote driving end and includes: After establishing a communication connection with a cloud platform, receiving video stream data uploaded in real time by a vehicle end from the cloud platform; wherein, the video stream data includes: video data collected by the vehicle end and corresponding to at least four different orientations of the vehicle end; Identifying video data of at least four different orientations in the video stream data, synchronously rendering the video data of the at least four different orientations in a preset video window, displaying camera images of at least four different orientations of the vehicle end, and detecting a remote driving start instruction; After determining that the remote driving start instruction is received, controlling the remote driving end to enter a remote control mode.

[0006] In some embodiments, the video stream data is obtained by the vehicle end in the following manner: Pre-constructing a plurality of driving nodes and a plurality of streaming nodes; wherein, each of the streaming nodes establishes a subscription relationship with one of the driving nodes; Obtaining the acquisition data of camera devices deployed at at least four different orientations through the plurality of driving nodes, and distributing them to a preset buffer position in real time through independent communication channels; Obtain the acquisition data distributed by the drive nodes with corresponding subscription relationships from the preset buffer position through the multiple live streaming nodes, and push the received acquisition data to the cloud platform after encoding processing.

[0007] In some embodiments, the obtaining the acquisition data of the camera devices deployed in at least four different directions through the multiple drive nodes includes: Obtain the acquisition data of the camera devices deployed in at least four different directions through the multiple drive nodes according to a preset hierarchical rule.

[0008] In some embodiments, the preset hierarchical rule includes: classifying all camera devices into first-level devices and second-level devices according to the deployment orientation; Obtaining the acquisition data of the camera devices deployed in at least four different directions through the multiple drive nodes according to a pre-classified rule includes: Obtain the acquisition data of a single first-level device through a single drive node, and obtain the acquisition data of at least two second-level devices through a single drive node.

[0009] In some embodiments, the pushing the received acquisition data to the cloud platform after encoding processing includes: Perform real-time format conversion on the received acquisition data within the live streaming node, generate intermediate data that conforms to the hardware encoding specification, and store it in a preset cache queue; Sequentially take out the intermediate data from the preset cache queue for encoding processing, obtain video stream data, and push the video stream data to the cloud platform.

[0010] In some embodiments, the pushing the video stream data to the cloud platform includes: Package the video stream data using the WebRTC protocol, attach orientation identification information, and then push it to the cloud platform.

[0011] In some embodiments, the method further includes: After receiving an input remote driving control instruction, parse the remote driving control instruction into a vehicle control instruction and upload it to the cloud platform, so that the vehicle end receives the vehicle control instruction issued by the cloud platform and performs corresponding remote control operations.

[0012] A second aspect of the present application provides a vehicle remote driving control system, including: A vehicle end, configured to collect video data of at least four different directions in real time, and process it into video stream data and upload it to the cloud platform; A cloud platform for receiving video stream data uploaded in real time by the vehicle terminal and, after establishing communication with the remote driving terminal, sending the video stream data to the remote driving terminal in real time; A remote driving terminal for receiving, after establishing a communication connection with the cloud platform, the video stream data uploaded in real time by the vehicle terminal; wherein, the video stream data includes: video data corresponding to at least four different orientations of the vehicle terminal; identifying the video data of at least four different orientations in the video stream data, synchronously rendering the video data of the at least four different orientations in a preset video window, displaying the camera images of at least four different orientations of the vehicle terminal and detecting a remote driving start instruction; and after determining that the remote driving start instruction is received, controlling the remote driving terminal to enter a remote control mode.

[0013] In some embodiments, the video stream data is obtained by the vehicle terminal in the following manner: Constructing in advance a plurality of driving nodes and a plurality of streaming nodes; wherein, each of the streaming nodes establishes a subscription relationship with one of the driving nodes; Obtaining, by the plurality of driving nodes, the acquisition data of the camera devices deployed at at least four different orientations and distributing the acquisition data to a preset buffer location in real time through independent communication channels; Obtaining, by the plurality of streaming nodes, the acquisition data distributed by the driving nodes with corresponding subscription relationships from the preset buffer location, encoding the received acquisition data, and pushing the encoded data to the cloud platform.

[0014] A third aspect of the present application provides an electronic device, including: A processor; and A memory having executable code stored thereon, which, when executed by the processor, causes the processor to execute the method as described above.

[0015] The technical solution provided by the present application may include the following beneficial effects: The technical solution of the present application realizes the real-time acquisition, encoding, and pushing of at least four video streams by receiving and synchronously rendering the video stream data of at least four different orientations uploaded by the vehicle terminal, overcomes the limitation in the prior art that only one or two video streams can be supported based on a single-process architecture, thereby enabling real-time visualization of the vehicle's surrounding environment, and further enabling remote operators to synchronously obtain multi-dimensional environmental information of the vehicle such as front, rear, left, and right, meeting the real-time and comprehensiveness requirements of remote cockpit control during the process of autonomous driving.

[0016] Furthermore, in the technical solution of the present application, a separated design of a driving node and a streaming node is adopted at the vehicle end. By establishing a subscription relationship between the driving node and the streaming node, the decoupling of the data collection process and the encoding and pushing process is effectively realized, the data processing time is effectively reduced, and the data is transmitted through an independent communication channel, effectively avoiding data competition and improving the data transmission efficiency, and further improving the real-time performance of the video data transmission process.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. Brief Description of the Drawings

[0018] By describing the exemplary embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other objects, features and advantages of the present application will become more obvious. Among them, in the exemplary embodiments of the present application, the same reference numerals generally represent the same components.

[0019] Figure 1 is a schematic flowchart of the vehicle remote driving control method shown in the embodiments of the present application; Figure 2 is another schematic flowchart of the vehicle remote driving control method shown in the embodiments of the present application; Figure 3 is another schematic flowchart of the vehicle remote driving control method shown in the embodiments of the present application; Figure 4 is a schematic structural diagram of the vehicle remote driving control system shown in the embodiments of the present application; Figure 5 is a schematic working flowchart of the vehicle remote driving control system shown in the embodiments of the present application; Figure 6 is a schematic structural diagram of the electronic device shown in the embodiments of the present application. Detailed Description of the Embodiments

[0020] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0021] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "the", and "said" used in this application and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0022] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of this application, "a plurality" means two or more unless otherwise specifically defined.

[0023] In the related art, the video stream interaction process during autonomous driving is mainly based on a single-process architecture, usually only supporting pushing one or two video streams for collecting driver operation data and road data of the vehicle, resulting in being unable to be applied to application scenarios with real-time and comprehensive requirements for vehicle surrounding environment data. For example, it cannot meet the real-time and comprehensive requirements of remote cockpit control during autonomous driving, restricting the technological development of remote cockpit control during autonomous driving.

[0024] In view of the above problems, the embodiments of this application provide a vehicle remote driving control method, which can realize the real-time visualization of the vehicle's surrounding environment, enabling remote operators to synchronously obtain multi-dimensional environment information of the vehicle and meeting the real-time and comprehensive requirements of remote cockpit control during autonomous driving.

[0025] The technical solutions of the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0026] Figure 1 It is a schematic flowchart of the vehicle remote driving control method shown in the embodiments of this application.

[0027] See Figure 1 , the vehicle remote driving control method of this application, which is applied to the remote driving end, includes: S110, after establishing a communication connection with the cloud platform, receiving the video stream data uploaded in real time by the vehicle end from the cloud platform; wherein, the video stream data includes: video data collected by the vehicle end and corresponding to at least four different orientations of the vehicle end.

[0028] In this step, after the remote driving terminal establishes communication with the cloud platform, such as through network communication technologies like 5G, 4G, WiFi, etc., it receives the video stream data uploaded by the vehicle terminal from the cloud platform. Among them, the remote driving terminal can be a computer terminal equipped with a display device, an input device, and a network communication module, or it can also be a dedicated remote driving console. For example, the remote driving terminal can be a smart phone, a smart tablet, a computer device, or a parallel cockpit.

[0029] It should be understood that the vehicle terminal collects video data of the surrounding environment of the corresponding vehicle in real time. Among them, the video stream data can include video data of at least four different orientations of the vehicle terminal. Further, the video stream data can at least include video data of four different orientations: the front, rear, left, and right of the vehicle terminal. In actual application, the vehicle terminal can be equipped with more than four camera devices to provide a more comprehensive environmental perception ability. For example, the vehicle terminal can be configured with five camera devices, located in the front, rear, left, right, and roof of the vehicle respectively. Another example is that multiple camera devices with different or the same orientations can be configured in the front of the vehicle terminal.

[0030] S120, identify the video data of at least four different orientations in the video stream data, synchronously render the video data of at least four different orientations in a preset video window, display the camera images of at least four different orientations of the vehicle terminal, and detect the remote driving start command.

[0031] In this step, after receiving the video stream data, identify the video data corresponding to different orientations from the video stream data. After the identification is completed, synchronously render the video data corresponding to different orientations into the preset video window respectively. It should be understood that synchronous rendering means rendering the video data of different orientations into their respective corresponding video windows simultaneously, ensuring that the video images of each orientation are synchronous in time. After displaying the corresponding video images, detect whether there is an input remote driving start command at the same time.

[0032] Among them, the remote driving start command can be input externally to the system or automatically generated internally by the system. For example, external input to the system can refer to input by a user or other remote terminals. Another example is that internal automatic generation by the system can refer to when the system operating conditions meet the automatically takeover conditions set in advance according to the event level, an automatic remote driving start command is generated.

[0033] Among them, the orientation information corresponding to different video data can be identified based on the orientation identification information included in the video stream data. That is to say, each different video data in the video stream data will be attached with an orientation identification to indicate which orientation of the vehicle the video data corresponds to. The orientation identification information can be added before the vehicle terminal uploads the video stream data.

[0034] Among them, the preset video window can be an area preset on the preset interface (such as the control interface) of the remote driving end for displaying video images. The preset video window corresponds to the video data. In this application, the video data corresponding to each azimuth can be collected by a camera device, that is, the video data corresponding to each azimuth corresponds to one path of video data.

[0035] Furthermore, the preset video windows can be arranged according to a preset layout so that the driver can intuitively observe the environment around the vehicle. For example, the preset video windows can be distributed horizontally. For another example, the preset video windows can correspond to the azimuth layout of the video data so that different video windows can intuitively represent the corresponding azimuth positions.

[0036] As an example, the video window corresponding to the front azimuth can be set at the upper central position of the interface, the video window corresponding to the rear azimuth can be set at the lower central position of the interface, the left video window can be set on the left side of the interface, and the right video window can be set on the right side of the interface. This layout method is consistent with the driver's habit of observing the environment around the vehicle during actual driving, which helps to improve the intuitiveness and convenience of remote driving.

[0037] S130. After determining that the remote driving start instruction is received, control the remote driving end to enter the remote control mode.

[0038] In this step, after the remote driving end successfully displays the camera images of each azimuth of the vehicle end, the driver can input a remote driving start instruction to the remote driving end. After receiving the start instruction, the remote driving end enters the remote control mode for remotely controlling the vehicle end.

[0039] Among them, it can be realized by monitoring the input information of the input device associated with the remote driving end to confirm the input of the remote driving start instruction. The input device can include but is not limited to a keyboard, a mouse, a touch screen or a dedicated driving control device.

[0040] It can be understood that in the remote control mode, the remote driving end will continuously monitor the driver's operation input, convert these inputs into vehicle control instructions, and send the converted vehicle control instructions to the vehicle end through the cloud platform, so as to realize the corresponding remote control operation on the vehicle end. It should be understood that after the remote driving end enters the remote control mode, it can also continuously receive and display the video stream data uploaded by the vehicle end, so that the driver can make corresponding driving decisions in a timely manner according to the real-time environmental conditions around the vehicle.

[0041] In this embodiment, the vehicle remote driving control method of the present application realizes real-time acquisition, encoding, and pushing of at least four video streams by receiving and synchronously rendering video stream data uploaded by the vehicle end from at least four different orientations, overcoming the limitation in the prior art that only one or two video streams can be supported based on a single-process architecture. Thus, real-time visualization of the vehicle's surrounding environment can be achieved, and further, remote operators can synchronously obtain multi-dimensional environmental information of the vehicle, such as the front, rear, left, and right, to meet the real-time and comprehensive requirements of remote cockpit control during the automatic driving process.

[0042] Figure 2 It is another schematic flowchart of the vehicle remote driving control method shown in the embodiments of the present application.

[0043] See Figure 2 , the vehicle remote driving control method of the present application is applied to the remote driving end and includes: S210, after establishing a communication connection with the cloud platform, receiving video stream data uploaded by the vehicle end in real time from the cloud platform; wherein, the video stream data includes: video data collected by the vehicle end and corresponding to at least four different orientations of the vehicle end.

[0044] S220, identifying video data of at least four different orientations in the video stream data, synchronously rendering the video data of at least four different orientations in a preset video window, displaying camera images of at least four different orientations of the vehicle end, and detecting a remote driving start instruction.

[0045] S230, after determining that the remote driving start instruction is received, controlling the remote driving end to enter the remote control mode.

[0046] Among them, steps S210 to S230 are similar to steps S110 to S130. For the specific implementation process, refer to the content described above, and details will not be elaborated here.

[0047] S240, after receiving an input remote driving control instruction, parsing the remote driving control instruction into a vehicle control instruction and uploading it to the cloud platform for the vehicle end to receive the vehicle control instruction sent by the cloud platform and execute corresponding remote control operations.

[0048] In this step, after the remote driving end enters the remote control mode, when receiving an input remote driving control instruction, the remote driving control instruction is parsed into a vehicle control instruction for controlling the vehicle to execute corresponding control operations, and the vehicle control instruction is uploaded to the cloud platform; the cloud platform sends the vehicle control instruction to the corresponding vehicle end, so that the vehicle end executes corresponding remote control operations according to the vehicle control instruction sent by the cloud platform.

[0049] It should be understood that the vehicle control instruction can be used to control relevant driving behaviors at the vehicle end, where the driving behaviors include but are not limited to: rotating the steering wheel, stepping on the accelerator pedal, stepping on the brake pedal, so as to achieve the control of the direction or speed at the vehicle end.

[0050] Among them, after the remote driving end receives the input remote driving control instruction, it will parse the received remote driving control instruction into a standardized vehicle control instruction, and this parsing process includes but is not limited to: signal acquisition, digital conversion, and standardization processing. Among them, through the above parsing process, it can be seen that different types of control instructions are converted into a unified format, which is convenient for subsequent transmission and processing.

[0051] Among them, when the remote driving end uploads the vehicle control instruction to the cloud platform, it can also add necessary header information to each vehicle control instruction during the upload process, such as timestamp, serial number, instruction type, etc., so that the cloud platform and the vehicle end can correctly receive and process the vehicle control instruction.

[0052] In this embodiment, the vehicle remote driving control method of the present application synchronously renders and real-time transmits the multi-directional video of the vehicle end to the control instruction of the vehicle end, so that the remote operator can perform in-depth coordination between the operation decision-making based on the comprehensive vision and the vehicle execution action, effectively improving the control accuracy of the vehicle end in the remote control application scenario, and enabling the operator to intuitively obtain the vehicle orientation information and quickly complete the control input at the same time, thereby greatly reducing the operation complexity of remote driving.

[0053] Figure 3 It is another process schematic diagram of the vehicle remote driving control method shown in the embodiment of the present application. For the convenience of understanding the technical solution of the present application, the process of the vehicle end obtaining the video stream data is further described below.

[0054] See Figure 3 , in the present application, the video stream data is obtained by the vehicle end in the following ways, including: S310, pre-build a plurality of driving nodes and a plurality of streaming nodes; wherein, each streaming node establishes a subscription relationship with one of the driving nodes.

[0055] In this step, it is necessary to pre-build a plurality of driving nodes and streaming nodes at the vehicle end to realize the data acquisition and processing of a plurality of camera devices. Among them, the driving node is responsible for directly interacting with the camera device and obtaining the original data collected by the corresponding camera device; the streaming node is responsible for processing and encoding the original data and pushing the processed data to the cloud platform. Each streaming node establishes a subscription relationship with a driving node, that is, each streaming node receives and processes the data from the driving node with which it has a subscription relationship.

[0056] It should be understood that the establishment of the subscription relationship is achieved through system configuration. During the system initialization phase, each streaming node is associated with the corresponding driving node according to the preset configuration information. Of course, the subscription relationship between the streaming node and the driving node can be reconfigured according to actual needs.

[0057] Among them, each driving node corresponds to one or more camera devices, and each streaming node corresponds to one driving node.

[0058] Among them, the number of streaming nodes can be the same as or less than the number of driving nodes, and there is no limit here.

[0059] S320, obtain the acquisition data of the camera devices deployed in at least four different orientations through multiple driving nodes, and distribute them to the preset buffer location in real time through independent communication channels.

[0060] In this step, after the driving node starts, it communicates with the corresponding camera device to obtain the raw data collected by these devices. The camera devices are deployed in at least four different orientations of the vehicle, which can include the front, rear, left, and right. The raw data obtained by each driving node will be distributed to the preset buffer location in real time through independent communication channels.

[0061] It should be understood that the independent communication channels can mean that each driving node has its own dedicated data transmission path, and the data transmission between different driving nodes will not interfere with each other. This design can improve the parallelism and efficiency of data transmission, and reduce the delay and blockage in the data transmission process.

[0062] Among them, when the driving node receives the acquisition data (i.e., the raw data) of the camera device, it also respectively identifies the received acquisition data. That is to say, when the driving node obtains the acquisition data of the camera device, it adds corresponding identification information to the received data, so that the subsequent streaming node can know the orientation of the camera device corresponding to the video data being processed according to the identification information during the subscription, processing, etc. process, and thus can improve the rendering accuracy of the subsequent video stream data, such as accurately rendering the video stream data into the windows corresponding to the camera devices in different orientations.

[0063] Among them, the preset buffer location can be a memory area in the system dedicated to temporarily storing raw data. All driving nodes can store the obtained raw data in the same or different buffers. Among them, the preset buffer location can be set with a fixed size and implement data storage according to the first-in, first-out overwrite principle, so as to ensure that the latest data of all camera devices can be saved in the preset buffer location, and effectively avoid occupying too much system resources due to too much data stored in the preset buffer location.

[0064] In some embodiments, the acquisition data of camera devices deployed in at least four different orientations can be obtained by multiple drive nodes according to a preset hierarchical rule. That is to say, when obtaining the acquisition data of the camera devices, the drive nodes can operate according to the preset hierarchical rule. For example, the drive nodes can adopt different data acquisition strategies according to the level of the camera devices, such as the number and orientation of the camera devices corresponding to a single drive node.

[0065] It should be understood that in this embodiment, the camera devices on the vehicle side are not only deployed according to the orientation, but also classified according to a preset hierarchical rule. This hierarchical rule can be formulated based on factors such as the importance of the camera devices, the data acquisition frequency, and the data processing priority.

[0066] In some embodiments, the preset hierarchical rule may include: classifying all camera devices into first-level devices and second-level devices according to the deployment orientation; the preset hierarchical rule, including: classifying all camera devices into first-level devices and second-level devices according to the deployment orientation; obtaining the acquisition data of the camera devices deployed in at least four different orientations by multiple drive nodes according to the pre-classified rule, including: obtaining the acquisition data of a single first-level device by a single drive node, and obtaining the acquisition data of at least two second-level devices by a single drive node.

[0067] It can be known that the first-level devices may refer to the camera devices that are more important for remote driving, such as the front camera device or the rear camera device, because the front view or the rear view is the most critical for driving decisions, and each first-level device must be obtained by an independent single drive node. The second-level devices may refer to the camera devices that are less important for remote driving, such as the left or right camera devices, and a single drive node can obtain the acquisition data of multiple second-level devices.

[0068] The configuration and working mode of the drive nodes are different through the hierarchical rule. For the first-level devices, usually a dedicated drive node is configured, and this node is only responsible for obtaining and processing the data from this device. This one-to-one configuration method can ensure that the data of the first-level devices can be processed in a timely and sufficient manner, and there will be no delay or loss due to resource competition. For the second-level devices, a drive node is configured to process the data of at least two (i.e., multiple) devices at the same time. For example, a drive node may be responsible for obtaining and processing the data from two camera devices on the left and right at the same time. This one-to-many configuration method can save system resources and improve resource utilization efficiency.

[0069] Through the above differential configuration and processing strategies, different processing priorities can be formed for the data of the camera devices in different orientations, so as to realize the priority processing of key data, thereby improving the safety and reliability of remote driving.

[0070] S330 obtains the acquisition data distributed by the driving nodes with corresponding subscription relationships from a preset buffer position through multiple streaming nodes, and encodes the received acquisition data and then pushes it to the cloud platform.

[0071] In this step, the streaming nodes will obtain the acquisition data distributed by the driving nodes with which they have subscription relationships from the preset buffer position. After obtaining the original data, the streaming nodes will encode these data. The purpose of encoding is to convert the original data into a format suitable for network transmission, usually including operations such as compression and encapsulation. After the encoding is completed, the streaming nodes will push the processed data to the cloud platform.

[0072] It should be understood that since each streaming node has established a subscription relationship with a specific driving node, each streaming node will only process the data from that driving node and will not process the data from other driving nodes.

[0073] In some embodiments, the process of encoding the received acquisition data by the streaming nodes and then pushing it to the cloud platform may include the following steps: S331 performs real-time format conversion on the received acquisition data within the streaming node, generates intermediate data that conforms to the hardware encoding specification, and stores it in a preset buffer queue.

[0074] After the streaming nodes obtain the acquisition data distributed by the driving nodes from the preset buffer position, they first need to perform real-time format conversion on these data. It should be understood that the original data format output by the imaging device is inconsistent with the input format supported by the hardware encoder, and conversion is required for subsequent encoding processing. After the format conversion is completed, the generated intermediate data will be stored in a preset buffer queue for further waiting for processing.

[0075] Among them, the specific operations of format conversion include but are not limited to: color space conversion (such as converting from RGB to YUV), resolution adjustment, and frame rate adjustment. By the above operations, the original data is converted into intermediate data that conforms to the hardware encoding specification, so that the subsequent hardware encoding processing can be carried out efficiently.

[0076] Among them, the buffer queue can be a first-in-first-out (FIFO) data structure for temporarily storing the intermediate data to be encoded. The buffer queue can have a fixed space size, or the space size can be dynamically adjusted according to the data generation rate and encoding rate to ensure that the data will not be lost due to queue overflow and will not occupy too much system resources due to an overly large queue.

[0077] S332 sequentially extracts the intermediate data from the preset buffer queue for encoding processing, obtains video stream data, and pushes the video stream data to the cloud platform.

[0078] Use a hardware encoder or a software encoder to retrieve intermediate data from a preset buffer queue in a first-in, first-out order and perform encoding processing on this data. It should be understood that the purpose of the encoding processing is to convert the intermediate data into video stream data suitable for network transmission, which usually includes operations such as compression and encapsulation. After the encoding processing is completed, the obtained video stream data will be pushed to the cloud platform.

[0079] In some embodiments, the process of pushing the video stream data to the cloud platform may include: encapsulating the video stream data using the WebRTC protocol and attaching orientation identification information, and then pushing it to the cloud platform.

[0080] During the pushing process, the streaming node will first encapsulate the encoded video stream data to generate data packets that conform to the WebRTC protocol specification. During the encapsulation process, necessary header information such as timestamps and sequence numbers will be added to each data packet so that the receiving end can correctly parse and process these data packets. In addition to the standard WebRTC protocol header information, the streaming node will also attach orientation identification information to each data packet. The orientation identification information is used to indicate which orientation of the vehicle (such as the front, rear, left, or right) the video data in this data packet corresponds to. After the encapsulation and attaching of the orientation identification information are completed, the streaming node will push the data packets to the cloud platform through the WebRTC protocol.

[0081] Among them, WebRTC (Web Real-Time Communication) is a technology that supports web browsers to conduct real-time voice conversations or video conversations, and it can provide peer-to-peer data transmission capabilities. The streaming node uses the WebRTC protocol to push the video stream data to the cloud platform, which improves the real-time nature of the effective data transmission process, and thus makes this application more suitable for the application scenario of remote driving. The WebRTC protocol supports multiple transmission methods, such as UDP (User Datagram Protocol), TCP (Transmission Control Protocol), etc.

[0082] Among them, the orientation identification information can be the stream index number of the video stream data. For example, starting from 0, in this application, it is 0, 1, 2, 3. The stream index numbers corresponding to the video stream data of the four different orientations (front, rear, left, and right) are 0, 1, 2, 3 respectively. During rendering, the video stream data of the four different orientations (front, rear, left, and right) will be rendered in the windows corresponding to the front, rear, left, and right video data according to the stream index number.

[0083] In this embodiment, for the vehicle remote driving control method of the present application, a separate design of a driving node and a streaming node is adopted at the vehicle end. By establishing a subscription relationship between the driving node and the streaming node, the decoupling of the data acquisition process and the encoding and pushing process is effectively achieved, the data processing time is effectively reduced, and the data is transmitted through an independent communication channel, effectively avoiding data competition and improving the data transmission efficiency, and further improving the real-time performance of the video data transmission process.

[0084] Corresponding to the foregoing embodiment of the application function implementation method, the present application also provides a vehicle remote driving control system, an electronic device, and corresponding embodiments.

[0085] Figure 4 It is a schematic structural diagram of the vehicle remote driving control system shown in the embodiment of the present application.

[0086] See Figure 4 , the vehicle remote driving control system 400 of the present application includes a vehicle end 410, a cloud platform 420, and a remote driving end 430.

[0087] The vehicle end 410 is configured to collect video data of at least four different orientations in real time and process it into video stream data for uploading to the cloud platform.

[0088] The cloud platform 420 is configured to receive the video stream data uploaded in real time by the vehicle end, and after establishing communication with the remote driving end, send the video stream data to the remote driving end in real time.

[0089] The remote driving end 430 is configured to receive the video stream data uploaded in real time by the vehicle end from the cloud platform after establishing a communication connection with the cloud platform; wherein, the video stream data includes: video data corresponding to at least four different orientations of the vehicle end; identifying the video data of at least four different orientations in the video stream data, synchronously rendering the video data of at least four different orientations in a preset video window, and displaying the camera images of at least four different orientations of the vehicle end; after determining that a remote driving start instruction is received, controlling the remote driving end to enter a remote control mode.

[0090] In some embodiments, the video stream data is obtained by the vehicle end 410 in the following manner: Construct multiple driving nodes and multiple streaming nodes in advance; wherein, each streaming node establishes a subscription relationship with one of the driving nodes; Obtain the acquisition data of the camera devices deployed at at least four different orientations through multiple driving nodes, and distribute it to a preset buffer position in real time through mutually independent communication channels; Obtain the acquisition data distributed by the driving nodes with corresponding subscription relationships from the preset buffer position through multiple streaming nodes, and perform encoding processing on the received acquisition data and then push it to the cloud platform.

[0091] In some embodiments, the vehicle end 410 obtains the acquisition data of camera devices deployed in at least four different orientations through multiple drive nodes, including: Obtaining the acquisition data of camera devices deployed in at least four different orientations through multiple drive nodes according to a preset hierarchical rule.

[0092] In some embodiments, the preset hierarchical rule includes: classifying all camera devices into first-level devices and second-level devices according to the deployment orientation; The vehicle end 410 obtains the acquisition data of camera devices deployed in at least four different orientations through multiple drive nodes according to a pre-classified rule, including: Obtaining the acquisition data of a single first-level device through a single drive node, and obtaining the acquisition data of at least two second-level devices through a single drive node.

[0093] In some embodiments, the vehicle end 410 performs encoding processing on the received acquisition data and then pushes it to the cloud platform, including: Performing real-time format conversion on the received acquisition data within the push stream node, generating intermediate data that conforms to the hardware encoding specification, and storing it in a preset cache queue; Sequentially taking out the intermediate data from the preset cache queue for encoding processing to obtain video stream data and pushing it to the cloud platform.

[0094] In some embodiments, the vehicle end 410 pushes the video stream data to the cloud platform 420, including: Encapsulating the video stream data using the WebRTC protocol and attaching orientation identification information, and then pushing it to the cloud platform.

[0095] In some embodiments, the remote driving end 430 is further configured to receive an input remote driving control instruction, parse the remote driving control instruction into a vehicle control instruction, and upload it to the cloud platform 420, so that the vehicle end 410 receives the vehicle control instruction issued by the cloud platform 420 and performs corresponding remote control operations.

[0096] Figure 5 It is a schematic diagram of the working process of the vehicle remote driving control system shown in the embodiments of the present application. Taking the vehicle end having four camera devices, namely a front camera device, a rear camera device, a left camera device, and a right camera device, as an example, the working process of the vehicle remote driving control system of the present application will be further elaborated in detail.

[0097] See Figure 5 , the working process of the vehicle remote driving control system of the present application includes the following steps: S510. The vehicle terminal collects the video data of four camera devices in different directions in real time through three driving nodes (driving nodes 1, 2, and 3). Among them, driving node 1 collects the video data of the front camera device, driving node 2 collects the video data of the rear camera device, and driving node 3 collects the video data of the left and right camera devices. That is, the three driving nodes collect a total of four-way video data.

[0098] S520. Each driving node in the vehicle terminal publishes each frame of video data through the ros topic.

[0099] S530. Three streaming nodes (streaming nodes 1, 2, and 3) in the vehicle terminal subscribe to the data topics of the corresponding camera devices respectively. Among them, streaming node 1 subscribes to the data topic published by driving node 1 correspondingly, streaming node 2 subscribes to the data topic published by driving node 2 correspondingly, and streaming node 3 subscribes to the data topic published by driving node 4 correspondingly.

[0100] S540. The three streaming nodes in the vehicle terminal convert each frame of video data received from the BGR format to the YUV_I420 format, and put the converted YUV data frame into the video frame buffer queue for caching.

[0101] S550. The vehicle terminal takes out the data frame from the cache queue, encodes it into the video stream data in the H.264 format, and pushes it to the cloud platform using the WebRTC protocol.

[0102] S560. The remote driving terminal pulls four-way video stream data from the cloud platform in real time and renders them in four windows respectively.

[0103] S570. After the remote cockpit driver inputs the remote driving start instruction in the remote driving terminal, the remote driving terminal is controlled to enter the remote control mode. After the remote cockpit driver inputs the remote driving control instructions such as operating the steering wheel, brakes, and accelerator in the remote driving terminal, they are sent to the vehicle terminal through the cloud platform to achieve the control of the vehicle terminal.

[0104] In this embodiment, the vehicle remote driving control system of the present application realizes the real-time acquisition, encoding, and pushing of at least four-way video streams by receiving and synchronously rendering at least four video stream data in different directions uploaded by the vehicle terminal, overcomes the limitation in the prior art that only one or two video streams can be supported based on a single-process architecture, thereby enabling the real-time visualization of the vehicle's surrounding environment, and further enabling remote operators to synchronously obtain multi-dimensional environmental information such as the front, rear, left, and right of the vehicle, meeting the real-time and comprehensiveness requirements of remote cockpit control during the process of autonomous driving.

[0105] Regarding the device in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0106] Figure 6 It is a schematic structural diagram of an electronic device shown in an embodiment of the present application.

[0107] See Figure 6 , the electronic device 1000 includes a memory 1010 and a processor 1020.

[0108] The processor 1020 may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0109] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, the ROM may store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all of the instructions and data required by the processor during operation. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a removable storage device that is readable and / or writable, such as a compact disc (CD), read-only digital versatile disc (such as DVD-ROM, dual-layer DVD-ROM), read-only Blu-ray disc, super density disc, flash memory card (such as SD card, min SD card, Micro-SD card, etc.), magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and instantaneous electronic signals transmitted wirelessly or by wire.

[0110] Executable code is stored on the memory 1010, and when the executable code is processed by the processor 1020, it can cause the processor 1020 to execute some or all of the methods described above.

[0111] In addition, the method according to the present application can also be implemented as a computer program or computer program product, which includes computer program code instructions for executing some or all of the above steps of the method according to the present application.

[0112] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium), on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or cloud platform, etc.), it causes the processor to execute some or all of the steps of the above method according to the present application.

[0113] The embodiments of the present application have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, the practical application, or the improvement of the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A vehicle remote driving control method, applied to a remote driving terminal, characterized in that: include: After establishing a communication connection with the cloud platform, receiving video stream data uploaded by the vehicle end in real time from the cloud platform; wherein the video stream data includes: video data collected by the vehicle end and corresponding to at least four different directions of the vehicle end; Identify video data in at least four different directions in the video stream data, synchronously render the video data in at least four different directions in a preset video window, display the camera images in at least four different directions of the vehicle end, and detect a remote driving start instruction; After confirming that the remote driving start command has been received, the remote driving terminal is controlled to enter the remote control mode.

2. The method according to claim 1, characterized in that The video stream data is obtained by the vehicle end in the following manner: Pre-build multiple driver nodes and multiple streaming nodes; each streaming node establishes a subscription relationship with one of the driver nodes; Acquire the collected data of the camera devices deployed in at least four different positions through the multiple driving nodes, and distribute them to the preset buffer positions in real time through independent communication channels; The plurality of streaming nodes obtain the collected data distributed by the driving nodes with corresponding subscription relationships from the preset buffer positions, and encode the received collected data and push it to the cloud platform.

3. The method according to claim 2, characterized in that The acquiring of the collected data of the camera devices deployed in at least four different positions by the multiple driving nodes includes: The collected data of the camera devices deployed in at least four different directions are obtained through the multiple driving nodes according to the preset classification rules.

4. The method according to claim 3, characterized in that The preset classification rules include: classifying all camera devices into first-level devices and second-level devices according to their deployment orientations; Acquiring the collected data of the camera devices deployed in at least four different positions through the multiple driving nodes according to the pre-classification rules, including: The collected data of a single first-level device is obtained through a single driving node, and the collected data of at least two second-level devices are obtained through a single driving node.

5. The method according to claim 2, characterized in that: The step of encoding the received collected data and then pushing it to the cloud platform includes: Performing real-time format conversion on the received collected data in the streaming node, generating intermediate data that complies with hardware encoding specifications and storing the intermediate data in a preset cache queue; The intermediate data is sequentially taken out from the preset cache queue for encoding processing to obtain video stream data and push it to the cloud platform.

6. The method according to claim 5, characterized in that The pushing to the cloud platform includes: The video stream data is encapsulated using the WebRTC protocol and the location identification information is added before being pushed to the cloud platform.

7. The method according to any one of claims 1 to 6, characterized in that: The method further includes: After receiving the remote driving control command, the remote driving control command is parsed into a vehicle control command and uploaded to the cloud platform, so that the vehicle side receives the vehicle control command issued by the cloud platform and performs the corresponding remote control operation.

8. A vehicle remote driving control system, characterized in that: include: The vehicle side is used to collect video data from at least four different directions in real time, and process it into video stream data and upload it to the cloud platform; A cloud platform, for receiving the video stream data uploaded by the vehicle end in real time, and after establishing communication with the remote driving end, sending the video stream data to the remote driving end in real time; The remote driving terminal is used to establish a communication connection with the cloud platform, and then receive the video stream data uploaded by the vehicle terminal in real time from the cloud platform; wherein the video stream data includes: video data corresponding to at least four different directions of the vehicle terminal; identifying the video data of at least four different directions in the video stream data, synchronously rendering the video data of at least four different directions in a preset video window, displaying the camera images of at least four different directions of the vehicle terminal and detecting the remote driving start command; after determining that the remote driving start command has been received, controlling the remote driving terminal to enter the remote control mode.

9. The system according to claim 8, characterized in that The video stream data is obtained by the vehicle end in the following manner: Pre-build multiple driver nodes and multiple streaming nodes; each streaming node establishes a subscription relationship with one of the driver nodes; Acquire the collected data of the camera devices deployed in at least four different positions through the multiple driving nodes, and distribute them to the preset buffer positions in real time through independent communication channels; The plurality of streaming nodes obtain the collected data distributed by the driving nodes with corresponding subscription relationships from the preset buffer positions, and encode the received collected data and push it to the cloud platform.

10. An electronic device, characterized in that: include: processor; as well as A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 7.

Images