Method and apparatus for remote assistance

Abstract

According to embodiments of the present disclosure, methods and apparatuses for remote assistance are provided. The method includes presenting an assistance interface associated with an autonomous vehicle; and presenting, in the assistance interface, a multi-path video stream corresponding to a plurality of viewing perspectives of the autonomous vehicle based on video data received from the autonomous vehicle, wherein the multi-path video stream includes a first video stream corresponding to a forward viewing perspective and a second video stream corresponding to a rearward viewing perspective, the first video stream and the second video stream are generated based on a video stitching model associated with the autonomous vehicle, the video stitching model is constructed based on a plurality of images captured by a plurality of cameras deployed on the autonomous vehicle. In this way, embodiments of the present disclosure can improve the reliability of remote assistance, thereby improving the driving safety of the autonomous vehicle.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of autonomous driving, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for remote assistance. Background Technology

[0002] Autonomous driving is a technology that uses computers to replace or assist human drivers in perceiving the vehicle's surroundings, planning the vehicle's trajectory, and controlling the vehicle to reach a designated destination.

[0003] During the journey of an autonomous vehicle, unexpected factors may disrupt its operation. In such cases, efficiently providing remote assistance to autonomous vehicles is a crucial aspect of improving journey safety and reliability. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for remote assistance is provided. The method includes: presenting an assistance interface associated with an autonomous vehicle; and presenting, in the assistance interface, multiple video streams corresponding to multiple viewing angles of the autonomous vehicle based on video data received from the autonomous vehicle, wherein the multiple video streams include a first video stream corresponding to a forward-looking view and a second video stream corresponding to a rearward-looking view, the first and second video streams being generated based on a video stitching model associated with the autonomous vehicle, the video stitching model being constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

[0005] In a second aspect of this disclosure, an apparatus for remote assistance is provided. The apparatus includes: a first presentation module configured to present an assistance interface associated with an autonomous vehicle; and a second presentation module configured to present, in the assistance interface, multiple video streams corresponding to multiple viewing angles of the autonomous vehicle based on video data received from the autonomous vehicle, wherein the multiple video streams include a first video stream corresponding to a forward-looking view and a second video stream corresponding to a rearward-looking view, the first and second video streams being generated based on a video stitching model associated with the autonomous vehicle, the video stitching model being constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method of the first aspect.

[0009] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0012] Figures 2A to 2C Example interfaces for remote assistance according to some embodiments of this disclosure are shown;

[0013] Figure 3 A schematic diagram of video stream stretching according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A flowchart for remote assistance according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic structural block diagram of an example device for remote assistance according to certain embodiments of the present disclosure is shown; and

[0016] Figure 6 A block diagram of an apparatus capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0019] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0020] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0021] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0022] As briefly mentioned earlier, autonomous driving is a technology that uses computers to replace or assist human drivers in perceiving the vehicle's surroundings, planning its trajectory, and controlling it to reach its designated destination. During an autonomous vehicle's journey, unforeseen factors may disrupt its operation. In such cases, efficiently providing remote assistance to autonomous vehicles is crucial for improving safety and reliability.

[0023] Embodiments of this disclosure propose a scheme for remote assistance. According to various embodiments of this disclosure, an assistance interface associated with an autonomous vehicle can be presented; and based on video data received from the autonomous vehicle, multiple video streams corresponding to multiple viewing angles of the autonomous vehicle are presented in the assistance interface, wherein the multiple video streams include a first video stream corresponding to a forward-looking view and a second video stream corresponding to a rearward-looking view. The first and second video streams are generated based on a video stitching model associated with the autonomous vehicle, which is constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

[0024] In this way, embodiments of the present disclosure can present multiple video feeds of an autonomous vehicle in a remote device, facilitating assistance personnel to perceive the vehicle's surrounding environment. Therefore, embodiments of the present disclosure can improve the reliability of remote assistance, thereby enhancing the driving safety of autonomous vehicles.

[0025] Example Environment

[0026] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, an assistant 140 can interact directly with a remote device 120, or interact with the remote device 120 via an attached device. The remote device 120 can present a user interface 130 to the assistant 140 for viewing vehicle information or performing operations such as providing remote assistance.

[0027] The remote device 120 may include, for example, a cloud device or an edge computing device. In some embodiments, such a remote device 120 may provide instructions to the vehicle 101 regarding remote assistance, thereby triggering the control system of the autonomous vehicle 101 to control the autonomous vehicle 101 based on the instructions.

[0028] Environment 100 may include an autonomous vehicle 101. In some embodiments, the autonomous vehicle 101 may be any type of vehicle capable of carrying people and / or objects and moving via a power system such as an engine, including but not limited to cars, trucks, buses, electric vehicles, motorhomes, etc. The autonomous vehicle 101 may be an automated driving vehicle (also referred to as an autonomous vehicle) that integrates functions such as environmental perception, planning and decision-making, and multi-level assisted driving.

[0029] like Figure 1 As shown, the autonomous vehicle 101 may be equipped with electronic devices 110. Electronic devices 110 may, for example, be communicatively connected to a remote device 120. For instance, electronic devices 110 may communicate with the remote device 120 via an appropriate wireless communication method. Electronic devices 110 can control the autonomous vehicle 101 according to target commands issued by the remote device 120.

[0030] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0031] Example interaction process

[0032] Figure 2 illustrates an example graphical interface 200 applied to a remote device 120 according to some embodiments of the present disclosure. The graphical interface 200 may be derived from a reference... Figure 1 A specific example of the user interface 130 provided by the described remote device 120 is given below. The interaction process of a passenger controlling the autonomous vehicle 101 through the user interface 130 will be described below with reference to FIG2.

[0033] In some embodiments, the remote device 120 presents an assistance interface associated with the autonomous vehicle. For example, such as... Figure 2A As shown, after receiving the target information sent by the autonomous vehicle 101, the remote device 120 can display the autonomous vehicle's assistance interface 200A in the user interface 130. The target information may be, for example, a remote assistance request, a fault repair request, etc.

[0034] In some embodiments, the remote device 120 presents multiple video streams corresponding to multiple viewing angles of the autonomous vehicle in an assistance interface based on video data received from the autonomous vehicle. The multiple video streams include a first video stream corresponding to a forward-looking view and a second video stream corresponding to a rearward-looking view. The first and second video streams are generated based on a video stitching model associated with the autonomous vehicle. The video stitching model is constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

[0035] As an example, after receiving video data sent by the autonomous vehicle 101, the remote device 120 can present a forward video stream 210 (i.e., the first video stream) and a backward video stream 211 (i.e., the second video stream) on the assistance interface 200A. The forward video stream 210 and the backward video stream 211 are stitched together from multiple forward and backward view videos collected by the autonomous vehicle 101. This allows for the presentation of more environmental information about the autonomous vehicle on the assistance interface, thereby improving the safety and reliability of the autonomous vehicle during operation.

[0036] In some scenarios, the remote device 120 can also display a vehicle status component 230 and a map component 240 in the assistance interface 200A. The vehicle status component 230 includes information such as the current speed and steering information of the autonomous vehicle 101. The map component 240 is displayed by the remote device 120 based on the perception information sent by the autonomous vehicle 101. The map component 240 includes graphic elements 245 for indicating the autonomous vehicle 101.

[0037] In some embodiments, the multiple video streams also include a third video stream corresponding to the top-down view of the autonomous vehicle. As an example, such as... Figure 2A As shown, the remote device 120 can also display a surround view video 212 (i.e., a third video stream) in the assistance interface 200A.

[0038] In some embodiments, the video stitching model includes a 3D bowl-shaped model associated with the autonomous vehicle, and the first and second video streams are generated based on the 3D bowl-shaped model and different virtual camera positions. The 3D bowl-shaped model is constructed by creating a 3D bowl-shaped region around the autonomous vehicle and then mapping the world coordinate system to the camera coordinate system to construct the virtual viewpoint.

[0039] As an example, the autonomous vehicle 101 can use a 3D bowl-shaped model to stitch together mid-range forward-facing videos from multiple perspectives to obtain a stitched video stream (i.e., the first video stream). By stitching together mid-range forward-facing videos, the resolution of the stitched video stream can be effectively improved.

[0040] In some embodiments, because the forward video stream is most helpful for assisting personnel in providing remote assistance, the first video stream is displayed in a larger size than the other video streams in the assistance interface. As an example, such as... Figure 2A As shown, remote device 120 can present forward video stream 210 in the assistance interface 200A at a size larger than other videos.

[0041] In some embodiments, the remote device 120 may, in response to the autonomous vehicle performing a target action, present an invoked video stream in the assistance interface.

[0042] As an example, such as Figure 2B As shown, when the autonomous vehicle 101 performs a target action, the remote device 120 can display and invoke video stream 220 in the assistance interface 200B. The target action can be, for example, a steering action, an acceleration action, a deceleration action, and a braking action. In some scenarios, the target action can also indicate that the autonomous vehicle 101 is in a trapped state, such as when the autonomous vehicle 101 has been scratched or impacted.

[0043] For example, when the autonomous vehicle 101 makes a left turn, the video stream 220 can be activated to display a video of the left side of the autonomous vehicle 101, to better assist the assistant 120 in perceiving the left turn process of the autonomous vehicle 101. As another example, when the autonomous vehicle 101 is struck, the video stream 220 can be activated to display a video including images of the impacted portion of the autonomous vehicle 101.

[0044] In some embodiments, the remote device 120 may, in response to receiving a viewing request on the assistance interface, present an invoked video stream in the assistance interface. As an example, such as Figure 2BAs shown, after receiving the selection of the view increase control 225 by the assistant 140, the remote device 120 can display the video stream 220 in the assistance interface 200B.

[0045] In some embodiments, the remote device 120 may, in response to receiving a remote assistance operation associated with the autonomous vehicle on the assistance interface, present an invoked video stream in the assistance interface.

[0046] As an example, such as Figure 2B As shown, the remote device 120 can display and invoke video stream 220 in the assistance interface 200B based on the remote assistance operation of the assistant 140. The remote assistance operation can be, for example, the assistant 140 selecting any autonomous vehicle control control, such as a control for triggering the autonomous vehicle to steer.

[0047] In some scenarios, when the remote device 120 receives the selection of the U-turn control by the assistant 140 (i.e., the assistance operation), the remote device 120 can display the rear video of the autonomous vehicle 101 in the assistance interface 200B (i.e., invoke the video stream) so that the assistant can observe the vehicle approaching from behind.

[0048] In other scenarios, when the remote device 120 receives the selection of the acceleration control by the assistant 140, the remote device 120 can display the forward video of the autonomous vehicle 101 in the assistance interface 200B (i.e., invoke the video stream).

[0049] In some embodiments, the viewpoint corresponding to the invoked video stream is determined based on the action type of the target action or the operation type of the remote assistance operation. For example, when the autonomous vehicle 101 makes a left turn, the invoked video will be a video from the left side of the autonomous vehicle 101, to more clearly present the left turn process. In this way, the remote device can present a more accurate video of the autonomous vehicle to the assistant 140, thereby providing the assistant with more accurate vehicle information.

[0050] In some embodiments, the remote device 120 may display interface elements associated with the autonomous vehicle in a first video stream based on messages from the master vehicle. The interface elements indicate trajectory information and perception information of the autonomous vehicle.

[0051] Continue to refer to Figure 2A The remote device 120 can display the driving trajectory 250 of the autonomous vehicle 101 in the first video stream 210. In some scenarios, the remote device 120 can also mark obstacles appearing in the first video stream 210 based on the perception information of the autonomous vehicle 101. For example, it can select obstacles appearing in the first video stream 210 to remind the assistant 140 that there are obstacles ahead.

[0052] In some embodiments, the remote device 120 may adjust the display size, display position, and / or aspect ratio of the multiple video streams in the assistance interface in response to receiving an adjustment operation of the multiple video streams. As an example, the remote device 120 may adjust the size and position of the invoked video 220 in the assistance interface based on the adjustment operation. For example, it may move the invoked video 220 to the lower left corner of the assistance interface.

[0053] In some embodiments, the remote device 120 may acquire a set of video frames associated with the first video stream. Further, the remote device 120 may stretch the set of video frames in the height direction. Finally, the remote device 120 may present the stretched set of video frames as the first video stream in an assistance interface.

[0054] As an example, remote device 120 can stretch a set of videos associated with the first video stream in the height direction to present them in the assistance interface as follows: Figure 3 The first video stream 213 after stretching is shown.

[0055] In some embodiments, the second video stream is overlaid on top of the first example stream. For example, as shown... Figure 2C As shown, the remote device 120 can overlay the second video stream 211 on top of the stretched first video stream 213. In this way, the screen space of the stretched first video stream 213 can be effectively utilized.

[0056] In some embodiments, the remote device 120 can determine the stretching ratios corresponding to multiple regions of a set of video frames, where the multiple regions correspond to different height ranges. Further, the remote device 120 can stretch multiple regions of a set of video frames based on multiple stretching ratios.

[0057] As an example, such as Figure 3 As shown, Figure 3 A schematic diagram of a stretched video according to an embodiment of this disclosure is shown. The remote device 120 can divide a first group of video frames into multiple regions, such as a first region 310 and a second region 320. The first region 310 is associated with a road scene. For example, if the original video height is 'a' and the adjusted height is 'b', then the height needs to be stretched by h = 'ba'. The height of the first region 310 is 2 / 3a. The height of the second region 320 is 1 / 3a. To ensure that the stretched road scene is as similar as the captured road scene, the height of the first region 310 can be stretched to 2 / 3a + 1 / 6h, and the height of the second region can be stretched to 1 / 3a + 1 / 6h.

[0058] In this way, by stretching the aspect ratio of the forward-looking video, other videos can be superimposed on the top area of ​​the forward-looking video. This ensures the clarity of the field of view that remote assistants need to focus on, while avoiding interference from other videos.

[0059] Based on the process described above, embodiments of this disclosure can support the presentation of multiple video feeds of autonomous vehicles on remote devices during autonomous vehicle operation. Therefore, embodiments of this disclosure can improve the handling capabilities in emergency scenarios, thereby enhancing the reliability of the travel services provided by autonomous vehicles.

[0060] Example process

[0061] Figure 4 A flowchart of an example process 400 for remote assistance according to some embodiments of the present disclosure is shown. Process 400 can be implemented at a remote device 120. References are made below. Figure 1 Describe the process 400.

[0062] like Figure 4 As shown, in box 410, remote device 120 presents an assistance interface associated with the autonomous vehicle.

[0063] In box 420, the remote device 120 presents multiple video streams corresponding to multiple viewing angles of the autonomous vehicle in the assistance interface based on the video data received from the autonomous vehicle. The multiple video streams include a first video stream corresponding to the forward-looking view and a second video stream corresponding to the rearward-looking view. The first and second video streams are generated based on a video stitching model associated with the autonomous vehicle. The video stitching model is constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

[0064] In some embodiments, the multiple video streams also include a third video stream corresponding to the top-down view of the autonomous vehicle.

[0065] In some embodiments, the video stitching model includes a three-dimensional bowl-shaped model associated with the autonomous vehicle, and the first and second video streams are generated based on the three-dimensional bowl-shaped model and different virtual camera positions.

[0066] In some embodiments, the first video stream is displayed in a larger size than the other video streams in the assistance interface.

[0067] In some embodiments, process 400 further includes: presenting a video stream in the assistance interface in response to the autonomous vehicle performing a target action; presenting a video stream in the assistance interface in response to receiving a viewing request in the assistance interface; and presenting a video stream in the assistance interface in response to receiving a remote assistance operation associated with the autonomous vehicle in the assistance interface.

[0068] In some embodiments, the viewpoint corresponding to the video stream is determined based on the action type of the target action or the operation type of the remote assistance operation.

[0069] In some embodiments, process 400 further includes: displaying interface elements associated with the autonomous vehicle in a first video stream based on messages from the autonomous vehicle.

[0070] In some embodiments, the interface elements indicate at least one of the following: trajectory information of the autonomous vehicle; perception information of the autonomous vehicle.

[0071] In some embodiments, process 400 further includes: adjusting the display size, display position and / or aspect ratio of the multiple video streams in the assistance interface in response to receiving an adjustment operation for the multiple video streams.

[0072] In some embodiments, process 400 further includes: acquiring a set of video frames associated with a first video stream; stretching the set of video frames in the height direction; and presenting the stretched set of video frames in an assistive interface as the first video stream.

[0073] In some embodiments, the second video stream is overlaid on top of the first video stream.

[0074] In some embodiments, stretching a set of video frames in the height direction includes: determining a plurality of stretching ratios corresponding to a plurality of regions of the set of video frames, the plurality of regions corresponding to different height ranges; and stretching the plurality of regions of the set of video frames based on the plurality of stretching ratios.

[0075] Example devices and equipment

[0076] Figure 5 A schematic structural block diagram of a device 500 for remote assistance according to certain embodiments of the present disclosure is shown. The device 500 may be implemented as or included in a remote device 120. Various modules / components in the device 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] As shown in the figure, the device 500 includes a first presentation module 510 configured to present an assistance interface associated with the autonomous vehicle; and a second presentation module 520 configured to present multiple video streams corresponding to multiple viewing angles of the autonomous vehicle in the assistance interface based on video data received from the autonomous vehicle. The multiple video streams include a first video stream corresponding to the forward-looking view and a second video stream corresponding to the rearward-looking view. The first and second video streams are generated based on a video stitching model associated with the autonomous vehicle. The video stitching model is constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

[0078] In some embodiments, the multiple video streams also include a third video stream corresponding to the top-down view of the autonomous vehicle.

[0079] In some embodiments, the video stitching model includes a three-dimensional bowl-shaped model associated with the autonomous vehicle, and the first and second video streams are generated based on the three-dimensional bowl-shaped model and different virtual camera positions.

[0080] In some embodiments, the first video stream is displayed in a larger size than the other video streams in the assistance interface.

[0081] In some embodiments, the device 500 further includes a video invocation presentation module configured to present an invocation video stream in an assistance interface in response to the autonomous vehicle performing a target action; to present an invocation video stream in an assistance interface in response to receiving a viewing request in the assistance interface; and to present an invocation video stream in an assistance interface in response to receiving a remote assistance operation associated with the autonomous vehicle in the assistance interface.

[0082] In some embodiments, the viewpoint corresponding to the video stream is determined based on the action type of the target action or the operation type of the remote assistance operation.

[0083] In some embodiments, the device 500 further includes an interface element display module configured to display interface elements associated with the autonomous vehicle in a first video stream based on messages from the autonomous vehicle.

[0084] In some embodiments, the interface elements indicate at least one of the following: trajectory information of the autonomous vehicle; perception information of the autonomous vehicle.

[0085] In some embodiments, the device 500 further includes an adjustment module configured to adjust the display size, display position, and / or aspect ratio of the multiple video streams in the assistance interface in response to receiving an adjustment operation for the multiple video streams.

[0086] In some embodiments, the apparatus 500 further includes a stretching module configured to acquire a set of video frames associated with a first video stream; stretch the set of video frames in the height direction; and present the stretched set of video frames in an assist interface as the first video stream.

[0087] In some embodiments, the second video stream is overlaid on top of the first video stream.

[0088] In some embodiments, the stretching module is further configured to: determine multiple stretching ratios corresponding to multiple regions of a set of video frames, the multiple regions corresponding to different height ranges; and stretch multiple regions of a set of video frames based on the multiple stretching ratios.

[0089] Figure 6A block diagram is shown illustrating a computing device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 6 The computing device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The computing device 600 shown can be used to implement Figure 1 Remote device 120.

[0090] like Figure 6 As shown, computing device 600 is in the form of a general-purpose computing device. Components of computing device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage devices 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 600.

[0091] Computing device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to computing device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within computing device 600.

[0092] The computing device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0093] The communication unit 640 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 600 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0094] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 600 can also communicate as needed with one or more external devices (not shown) via communication unit 640. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 600, or with any device (e.g., network card, modem, etc.) that enables computing device 600 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0095] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0096] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0098] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0100] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for remote assistance, comprising: Presents an assistance interface associated with autonomous vehicles; as well as Based on the video data received from the autonomous vehicle, the assistance interface presents multiple video streams corresponding to multiple viewing angles of the autonomous vehicle. The multiple video streams include a first video stream corresponding to the forward-looking view and a second video stream corresponding to the rearward-looking view. The first video stream and the second video stream are generated based on a video stitching model associated with the autonomous vehicle. The video stitching model is constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

2. The method according to claim 1, wherein the multi-video stream further includes a third video stream corresponding to the top-down view of the autonomous vehicle.

3. The method of claim 1, wherein the video stitching model includes a three-dimensional bowl-shaped model associated with the autonomous vehicle, and the first video stream and the second video stream are generated based on the three-dimensional bowl-shaped model and different virtual camera positions.

4. The method according to claim 1, wherein the display size of the first video stream in the assistance interface is larger than that of the other video streams in the multi-video stream.

5. The method according to claim 1, further comprising: In response to the autonomous vehicle performing the target action, a video stream is displayed in the assistance interface; In response to receiving a viewing request on the assistance interface, an initiation video stream is presented on the assistance interface; In response to receiving a remote assistance operation associated with the autonomous vehicle on the assistance interface, a video stream is invoked on the assistance interface.

6. The method according to claim 5, wherein the viewpoint corresponding to the invoked video stream is determined based on the action type of the target action or the operation type of the remote assistance operation.

7. The method according to claim 1, further comprising: Based on messages from the autonomous vehicle, interface elements associated with the autonomous vehicle are displayed in the first video stream.

8. The method of claim 7, wherein the interface element indicates at least one of the following: The trajectory information of the autonomous vehicle; The perception information of the autonomous vehicle.

9. The method according to claim 1, further comprising: In response to receiving an adjustment operation for the multiple video streams, the display size, display position, and / or aspect ratio of the multiple video streams in the assistance interface are adjusted.

10. The method according to claim 1, further comprising: Obtain a set of video frames associated with the first video stream; Stretch the set of video frames in the height direction; as well as The stretched set of video frames is presented in the assistance interface as the first video stream.

11. The method of claim 10, wherein the second video stream is superimposed on top of the first video stream.

12. The method of claim 10, wherein stretching the set of video frames in the height direction comprises: Determine multiple stretch ratios corresponding to multiple regions of the set of video frames, wherein the multiple regions correspond to different height ranges; as well as Based on the multiple stretching ratios, the multiple regions of the set of video frames are stretched.

13. An apparatus for remote assistance, comprising: The first presentation module is configured to present an assistance interface associated with the autonomous vehicle; The second presentation module is configured to present multiple video streams corresponding to multiple viewing angles of the autonomous vehicle in the assistance interface based on video data received from the autonomous vehicle. The multiple video streams include a first video stream corresponding to the forward-looking view and a second video stream corresponding to the rearward-looking view. The first and second video streams are generated based on a video stitching model associated with the autonomous vehicle. The video stitching model is constructed based on multiple images captured by multiple cameras deployed on the autonomous vehicle.

14. An electronic device comprising: At least one processing unit; as well as At least one memory is coupled to at least one processing unit and stores instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 12.

16. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 12.

Images