A vehicle networking cloud-based multi-vehicle cooperative panoramic video recording method and system

Abstract

The application provides a kind of multi-vehicle coordination panoramic video method and system based on Internet of Vehicles cloud, belong to intelligent networked vehicle technical field, the method includes: in response to the panoramic video session request initiated by target vehicle, obtain the position coordinates of target vehicle, desired shooting radius;Search the several vehicles as collaborative vehicles in the search radius of the position coordinates as the center within in the circle;According to the field of view angle of the camera of target vehicle and each collaborative vehicle, determine the pitch angle and yaw angle, and issue to target vehicle and each collaborative vehicle, to ensure that adjacent vehicle field of view angle edge overlap forms seamless surround view;Obtain the video stream and pose data photographed by the camera of target vehicle and each collaborative vehicle, three-dimensionally fuse the video stream and pose data, obtain 360° three-dimensional panoramic video file.The application can automatically generate 360° three-dimensional panoramic video by Internet of Vehicles cloud+multi-vehicle coordination using on-site vehicles, effectively solve the single-vehicle visual angle blind area problem of prior art.

Description

Technical Field

[0001] This invention relates to the field of intelligent connected vehicle technology, and in particular to a multi-vehicle collaborative panoramic video recording method and system based on vehicle network cloud. Background Technology

[0002] Existing car dashcams can only provide single-vehicle view footage, which is limited and cannot cover the entire surrounding area in open scenes such as camping or car club gatherings. Drone aerial photography is limited by airspace control, battery life, and noise; handheld panoramic cameras require specialized operation and are subject to altitude restrictions.

[0003] Another existing technology involves multi-vehicle image-BEV feature-level fusion. Each vehicle converts its front view image into a bird's-eye view (BEV) feature map, and the central vehicle uses element-wise max to extract the maximum value pixel by pixel, resulting in a fused feature map that retains the image from which the view is clearest. However, this technology focuses on autonomous driving sensing, not entertainment recording, and is only applicable to driving / intersection scenarios. It only supports feature-level fusion and cannot achieve 3D reconstruction.

[0004] Therefore, there is an urgent need for a solution that can automatically generate 360° three-dimensional panoramic videos without the need for additional equipment and using existing vehicles on site. Summary of the Invention

[0005] This invention proposes a closed-loop recording solution of "vehicle network cloud + multi-vehicle collaboration + blockchain evidence storage", which aims to solve at least one of the above-mentioned existing technical problems.

[0006] In a first aspect, embodiments of the present invention provide a multi-vehicle collaborative panoramic video recording method based on vehicle network cloud, including: In response to a panoramic video recording session request initiated by the target vehicle, obtain the target vehicle's location coordinates and desired shooting radius; The search radius is set according to the desired shooting radius, and several vehicles that are online and whose cameras are idle are retrieved within the search radius centered at the location coordinates as cooperative vehicles; The pitch and yaw angles of the cameras of the target vehicle and each cooperating vehicle are determined based on the field of view of the cameras of the target vehicle and each cooperating vehicle, and then sent to the target vehicle and each cooperating vehicle to ensure that the edges of the field of view of adjacent vehicles overlap to form a seamless surround view. The video streams and pose data captured by the cameras of the target vehicle and each cooperating vehicle are acquired, and the video streams and pose data are fused in three dimensions to obtain a 360° three-dimensional panoramic video file.

[0007] In a preferred embodiment, it further includes: The hash of the 3D panoramic video file is calculated using a hash function, written to the blockchain platform, and the IPFS content identifier is returned to the target vehicle and authorized user terminal.

[0008] In a preferred embodiment, the step of setting a search radius based on the desired shooting radius and retrieving several online vehicles with idle cameras within the search radius centered at the location coordinates as cooperating vehicles includes: Set the search radius to be greater than or equal to the desired shooting radius based on the desired shooting radius; Within a search radius centered at the location coordinates, N vehicles that are online and whose cameras are idle are retrieved as cooperative vehicles, where N is greater than or equal to 3; When N is greater than 3, the vehicles that maximize complementary perspectives and minimize overlapping redundancy are selected as cooperative vehicles based on the ground projection area of ​​the camera fields of view of each cooperative vehicle.

[0009] In a preferred embodiment, the step of determining the pitch and yaw angles of the target vehicle and each cooperating vehicle's cameras based on their field of view angles, and then sending this information to the target vehicle and each cooperating vehicle to ensure that the edges of the field of view angles of adjacent vehicles overlap to form a seamless surround view: Ensure that the overlap ratio of the field of view edges of adjacent vehicles is 20% ± 3%.

[0010] In a preferred embodiment, the step of acquiring video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and performing three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file includes: Acquire video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle; Construct an initial graph optimization model using pose data; Sparse point clouds are recovered from video streams using structure-reconstruction-motion technique; Lightweight neural radiation field technology is used for interpolation of blind areas; An 8K×4K 360° 3D panoramic video file is generated using a surface reconstruction algorithm.

[0011] In a preferred embodiment, the steps of calculating the hash of the 3D panoramic video file using a hash function, writing it to the blockchain platform, and returning the IPFS content identifier to the target vehicle and authorized user terminal are as follows: The SHA-256 hash of the 3D panoramic video file is calculated, and the smart contract is called to write it to the Ethereum sidechain, returning the IPFS content identifier to the target vehicle and authorized user terminal.

[0012] In a preferred embodiment, the location coordinates are GNSS coordinates; The video streams captured by the cameras of the target vehicle and each cooperating vehicle are captured frame-level synchronously by the cameras of the target vehicle and each cooperating vehicle through NTP time synchronization, and are uploaded to the cloud through a 5G uRLLC link after being encoded using the H.265 standard.

[0013] In a second aspect, embodiments of the present invention provide a multi-vehicle collaborative panoramic recording system based on a vehicle network cloud, configured to implement any of the methods described in the first aspect, the system comprising: The triggering module is used to respond to the panoramic recording session request initiated by the target vehicle and obtain the target vehicle's position coordinates and expected shooting radius; The collaborative vehicle screening module is used to set a search radius according to the desired shooting radius, and to retrieve several vehicles that are online and whose cameras are idle within the search radius centered on the location coordinates as collaborative vehicles. The adaptive viewing angle adjustment module is used to determine the pitch angle and yaw angle of the target vehicle and each cooperative vehicle's camera based on the field of view angle of the target vehicle and each cooperative vehicle's camera, and send them to the target vehicle and each cooperative vehicle to ensure that the edges of the field of view angles of adjacent vehicles overlap to form a seamless surround view. The spatiotemporal synchronization acquisition module is used to acquire video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and to perform three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the first aspect.

[0015] Fourthly, embodiments of the present invention provide a computer-readable medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0016] Beneficial effects of this invention: This invention solves the blind spot problem of single-vehicle perspective in existing technologies by automatically filling blind spots through multi-vehicle linkage.

[0017] This invention solves the problem of video being scattered and easily lost in existing technologies by using real-time cloud fusion and blockchain to effectively prevent tampering.

[0018] This invention effectively reduces the complexity of operation by using a one-click, fully automatic recording method, thus solving the problem of complex operation in existing technologies. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of a multi-vehicle collaborative panoramic video recording method based on vehicle network cloud provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic flowchart of an optional implementation of step S2 provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of an optional implementation of step S4 provided in an embodiment of the present invention.

[0022] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of another multi-vehicle collaborative panoramic video recording system based on vehicle network cloud provided in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the technical solutions of the present invention, exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Where there is no conflict, the various embodiments of the present invention and the features thereof may be combined with each other.

[0026] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0028] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined herein.

[0029] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, and disclosure of user personal information all comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations (e.g., the "Information Security Technology - Personal Information Security Specification"). For example: appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identity targeting is eliminated when using personal information to avoid precisely locating a specific individual.

[0030] Some abbreviations and key terms in this invention are defined as follows: V2X: Vehicle to Everything. It refers to the technology that enables vehicles to exchange information with their surroundings (other vehicles, road infrastructure, networks, etc.).

[0031] GNSS coordinates: Position coordinates calculated from signals from the Global Navigation Satellite System. GNSS is a general term that includes GPS (Global Positioning System), BDS (BeiDou), etc.

[0032] IMU: Inertial Measurement Unit, is a sensor that can sense the motion state of an object in real time. It directly outputs linear acceleration and angular velocity data through a combination of a three-axis accelerometer and a gyroscope. These raw data are processed by algorithms to obtain pose data, that is, the position and orientation (direction) of the object in three-dimensional space.

[0033] IPFS: InterPlanetary File System, is a distributed storage and sharing protocol that uses content addressing instead of traditional IP address addressing, making the network more open, faster, and more secure.

[0034] CID: Content Identifier in IPFS, is a file's "digital fingerprint" that is generated based on the content itself, rather than depending on the storage location. This ensures the integrity and uniqueness of the data.

[0035] Ethereum: Ethereum is an open-source, decentralized blockchain platform that allows developers to build and run decentralized applications (DApps) and smart contracts on it.

[0036] H.265 encoding is the High Efficiency Video Coding (HEVC) standard.

[0037] SHA-256 is the 256-bit version of the Secure Hash Algorithm, a widely used cryptographic hash function.

[0038] NTP (Network Time Protocol) is a precise method for automatically synchronizing computer system clocks over a network.

[0039] 5G uRLLC (Ultra-Reliable Low-Latency Communications) links refer to dedicated communication paths designed in 5G networks to meet the requirements of ultra-reliable and low-latency communication.

[0040] SfM: Structure-from-Motion, also known as Structure from Motion, is a technique that reconstructs 3D scenes by analyzing image sequences. Its core is to recover sparse 3D point clouds and camera motion parameters from 2D images.

[0041] NeRF, or Neural Radiation Field, is an advanced computer vision technology used for 3D scene reconstruction and novel perspective synthesis.

[0042] Poisson surface reconstruction is an algorithm for recovering the surface of an object from 3D point cloud data.

[0043] FOV: Field of View, is an inherent optical property of a camera that describes the range of angles that the camera can see.

[0044] Camera field of view: The physical space area actually captured by the camera in three-dimensional space is a solid pyramid formed by the opening of the FOV angle.

[0045] The projected area of ​​the camera's field of view on the ground is the two-dimensional region formed by the intersection of the camera's field of view cone and the ground plane.

[0046] Pitch angle: The rotation of the camera around the lateral axis (tilt up / tilt down).

[0047] Yaw angle: The rotation of the camera around its vertical axis (left / right turn).

[0048] In this embodiment, for ease of description, the cloud is used as the execution subject in the following description. The execution subject can also be a software module, or other electronic devices capable of performing the following functions.

[0049] Figure 1 This is a flowchart illustrating a multi-vehicle collaborative panoramic recording method based on a vehicle network cloud, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes: Step S1: In response to the panoramic video recording session request initiated by the target vehicle (initiating vehicle), obtain the position coordinates and expected shooting radius R of the target vehicle; Step S2: Set the search radius according to the desired shooting radius R, and search for several vehicles that are online and whose cameras are idle within the search radius centered on the position coordinates as cooperative vehicles; Step S3: Determine the pitch and yaw angles of the cameras of the target vehicle and each cooperating vehicle based on the field of view (FOV) of the cameras of the target vehicle and each cooperating vehicle, and send them to the target vehicle and each cooperating vehicle to ensure that the edges of the field of view (FOV) of adjacent vehicles overlap to form a seamless surround view. Step S4: Obtain video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and perform three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file.

[0050] This invention acquires video streams captured by cameras of the target vehicle and each cooperating vehicle through vehicle-to-everything (V2X) cloud and multi-vehicle collaboration. After three-dimensional fusion, a 360° three-dimensional panoramic video file can be obtained. No additional equipment is required, and the 360° three-dimensional panoramic video can be automatically generated using existing vehicles on site. Therefore, it can solve the blind spot problem of single-vehicle perspective in existing technologies.

[0051] In addition, in response to the initiation of the target vehicle, the present invention effectively reduces the complexity of operation by initiating with one click and recording the entire process automatically, and also solves the problem of complex operation in the prior art.

[0052] In some embodiments, the method further includes: step S5, calculating a hash for the 3D panoramic video file using a hash function, writing it to a blockchain platform, and returning the IPFS Content Identifier (CID) to the target vehicle (initiating vehicle) and the authorized user terminal.

[0053] This invention forms a closed-loop recording system through vehicle network cloud, multi-vehicle collaboration, and blockchain evidence storage, which can effectively prevent tampering and enable rapid sharing, solving the problem of video being scattered and easily lost in existing technologies.

[0054] In some embodiments, such as Figure 2As shown, step S2, which involves setting a search radius based on the desired shooting radius R, and retrieving several online vehicles with idle cameras within the search radius centered at the location coordinates as cooperating vehicles, includes: Step S21: Set the search radius to be greater than or equal to the expected shooting radius R (e.g., search radius = 2R) based on the expected shooting radius R. Step S22: Retrieve N vehicles that are online and whose cameras are idle within the search radius centered on the location coordinates as cooperative vehicles, where N is greater than or equal to 3; Step S23: When N is greater than 3, the vehicles that maximize complementary viewing angles and minimize overlapping redundancy are calculated based on the ground projection area of ​​the field of view of each cooperating vehicle's camera and are used as cooperating vehicles.

[0055] The solution that maximizes complementary perspectives and minimizes overlapping redundancy can be achieved using the following optimization function: max Σ(i=1→N) Ωi st Ωi∩Ωj ≤ σ , i≠j, Ωi is the projected area of ​​the field of view of the i-th vehicle's camera on the ground, and σ is the preset maximum allowed overlap area. The purpose of the above-mentioned optimized collaborative vehicle is to maximize the total area covered by all cameras to obtain more comprehensive environmental information, but also to limit the overlap area between any two camera fields of view to avoid information duplication and save communication and computing resources.

[0056] In some embodiments, step S3, which involves determining the pitch and yaw angles of the cameras of the target vehicle and each cooperating vehicle based on their field of view (FOV), and sending these angles to the target vehicle and each cooperating vehicle, ensures that the edges of the FOVs of adjacent vehicles overlap to form a seamless surround view. Ensure that the overlap ratio of the field of view (FOV) edges of adjacent vehicles is 20% ± 3%.

[0057] The overlap ratio is 20%±3% (17%≤overlap ratio≤23%), which refers to the proportion of overlapping parts in the field of view of two adjacent cameras. This ratio comprehensively considers requirements such as stitching quality, computational overhead, field of view efficiency, and calibration tolerance.

[0058] In some embodiments, such as Figure 3 As shown, step S4, which involves acquiring video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and then performing three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file, includes the following steps: Step S41: Obtain video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle; Step S42: Construct an initial graph optimization model using pose data; Step S43: Use Structure-from-Motion motion reconstruction technology to recover sparse point clouds from the video stream; Step S44: Lightweight NeRF neural radiation field technology is used to interpolate the blind spot. This automatic blind spot filling technology is also helpful in solving the blind spot problem of single vehicle view in existing technologies. Step S45: Generate an 8K×4K 360° three-dimensional panoramic video file using the Poisson surface reconstruction algorithm.

[0059] In some embodiments, step S5, which involves calculating a hash of the 3D panoramic video file using a hash function, writing it to the blockchain platform, and returning an IPFS Content Identifier (CID) to the target vehicle (initiating vehicle) and the authorized user terminal, is as follows: The SHA-256 hash of the 3D panoramic video file is calculated, and the smart contract is called to write it to the Ethereum sidechain, returning the IPFS Content Identifier (CID) to the target vehicle and authorized user terminal.

[0060] In some embodiments, the location coordinates are GNSS coordinates, including GPS, BDS (BeiDou), etc.; the video streams captured by the cameras of the target vehicle and each cooperating vehicle are captured by the cameras of the target vehicle and each cooperating vehicle through NTP time synchronization, and uploaded to the cloud through a 5G uRLLC link after being encoded using the H.265 standard.

[0061] Based on the same inventive concept, embodiments of the present invention also provide a multi-vehicle collaborative panoramic recording system based on a vehicle network cloud, configured to implement any of the methods described in the above embodiments, the system comprising: The triggering module is used to respond to the panoramic recording session request initiated by the target vehicle and obtain the target vehicle's position coordinates and expected shooting radius; The collaborative vehicle screening module is used to set a search radius according to the desired shooting radius, and to retrieve several vehicles that are online and whose cameras are idle within the search radius centered on the location coordinates as collaborative vehicles. The adaptive viewing angle adjustment module is used to determine the pitch angle and yaw angle of the target vehicle and each cooperative vehicle's camera based on the field of view angle of the target vehicle and each cooperative vehicle's camera, and send them to the target vehicle and each cooperative vehicle to ensure that the edges of the field of view angles of adjacent vehicles overlap to form a seamless surround view. The spatiotemporal synchronization acquisition module is used to acquire video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and to perform three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file.

[0062] Based on the same inventive concept, embodiments of the present invention also provide an electronic device. Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Figure 4 As shown, an embodiment of the present invention provides an electronic device including: one or more processors 101, a memory 102, and one or more I / O interfaces 103. The memory 102 stores one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the methods described in the above embodiments; the one or more I / O interfaces 103 are connected between the processor and the memory, configured to enable information interaction between the processor and the memory.

[0063] The processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read / write interface) 103 is connected between the processor 101 and the memory 102, and can realize information interaction between the processor 101 and the memory 102, including but not limited to a data bus (Bus).

[0064] In some embodiments, the processor 101, memory 102, and I / O interface 103 are interconnected via bus 104, and thus connected to other components of the computing device.

[0065] In some embodiments, the one or more processors 101 include a field-programmable gate array.

[0066] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable medium. This computer-readable medium stores a computer program, wherein, when executed by a processor, the program implements the steps of any of the methods described in the above embodiments. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

[0067] Figure 5 This is a schematic diagram of another multi-vehicle collaborative panoramic video recording system based on vehicle network cloud provided in an embodiment of the present invention. Figure 5 As shown, the system workflow is as follows: 1. Triggering phase: Vehicle 1 (30-1) initiates a "panoramic recording session" with one click through the vehicle human-machine interface. The vehicle-side V2X module uploads the session ID, GNSS coordinates, and desired shooting radius R to the vehicle network cloud (cloud server). 2. Collaborative Vehicle Selection: The cloud server searches for N≥3 collaborative vehicles (30-2 / 3 / 4) that are online and have idle cameras within a radius of 2R. With the objective of "maximizing complementary perspectives and minimizing overlapping redundancy," the following optimization function is solved: max Σ(i=1→N) Ωi st Ωi∩Ωj ≤ σ , i≠j, Where Ωi is the projected area of ​​the field of view of the camera of the i-th vehicle on the ground, and σ is the maximum allowed overlap area; 3. Adaptive viewing angle adjustment: After receiving the recommended pitch angle θi and yaw angle ψi from the cloud server, vehicle 1 and each cooperating vehicle complete the angle locking within 1 second through the vehicle-mounted gimbal or electronic clipping method, ensuring that the FOV edges of adjacent vehicles overlap by 20%±3% to form a seamless surround view; 4. Spatiotemporal synchronization acquisition: Vehicle 1 and each cooperating vehicle use NTP timing as the primary method and IMU clock as the backup to achieve frame-level synchronization. After H.265 encoding, the video stream and IMU pose data are uploaded in parallel through the 5G uRLLC link. 5. Cloud-based 3D Fusion: The cloud server first uses IMU pose to construct an initial image optimization model, then uses Structure-from-Motion to recover the sparse point cloud; lightweight NeRF interpolation is used for blind areas, and finally, 8K×4K 360° 3D panoramic video is generated through Poisson surface reconstruction. 6. Blockchain Evidence Storage: Calculate the SHA-256 hash of the panoramic video file, call the smart contract to write it to the Ethereum sidechain, and return the IPFS Content Identifier (CID) to the initiating vehicle and authorized user terminal to achieve tamper-proof and fast sharing.

[0068] In some embodiments, in a camping scenario, a method for completing 360° panoramic video recording using multi-vehicle collaboration includes: Step 1: The user clicks "Start Panoramic Recording" on the vehicle's infotainment system → The vehicle's infotainment system sends a RESTful request to the cloud server, carrying GPS coordinates and an estimated radius of 50 m; Step 2: The cloud server searches for online vehicles within a 200-meter radius, filters out 3 vehicles (A / B / C) with usable cameras and battery levels >30%, and returns the recommended tilt angle; Step 3: The user vehicle and vehicles A / B / C adjust the camera to the recommended angle through the vehicle domain controller, start synchronous recording at 30fps, and send back H.265+IMU data; Step 4: The cloud server uses NTP timestamp alignment, optimizes spatial registration based on SLAM pose graph, uses NeRF interpolation for blind spots, and finally generates 8K×4K 360° video; Step 5: Perform a SHA-256 hash on the video file, write it to the Ethereum sidechain, and return the IPFS link to the initiating user. The user can then scan the QR code with their mobile phone to play the video.

[0069] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0070] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0071] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0072] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0073] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0074] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0075] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0076] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0077] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0078] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multi-vehicle collaborative panoramic video recording method based on vehicle network cloud, characterized in that, include: In response to a panoramic video recording session request initiated by the target vehicle, obtain the target vehicle's location coordinates and desired shooting radius; The search radius is set according to the desired shooting radius, and several vehicles that are online and whose cameras are idle are retrieved within the search radius centered at the location coordinates as cooperative vehicles; The pitch and yaw angles of the cameras of the target vehicle and each cooperating vehicle are determined based on the field of view of the cameras of the target vehicle and each cooperating vehicle, and then sent to the target vehicle and each cooperating vehicle to ensure that the edges of the field of view of adjacent vehicles overlap to form a seamless surround view. The video streams and pose data captured by the cameras of the target vehicle and each cooperating vehicle are acquired, and the video streams and pose data are fused in three dimensions to obtain a 360° three-dimensional panoramic video file.

2. The method according to claim 1, characterized in that, Also includes: The hash of the 3D panoramic video file is calculated using a hash function, written to the blockchain platform, and the IPFS content identifier is returned to the target vehicle and authorized user terminal.

3. The method according to claim 1 or 2, characterized in that, The step of setting a search radius based on the desired shooting radius and retrieving several online vehicles with idle cameras within the search radius centered at the location coordinates as cooperating vehicles includes: Set the search radius to be greater than or equal to the desired shooting radius based on the desired shooting radius; Within a search radius centered at the location coordinates, N vehicles that are online and whose cameras are idle are retrieved as cooperative vehicles, where N is greater than or equal to 3; When N is greater than 3, the vehicles that maximize complementary perspectives and minimize overlapping redundancy are selected as cooperative vehicles based on the ground projection area of ​​the camera fields of view of each cooperative vehicle.

4. The method according to claim 3, characterized in that, In the step of determining the pitch and yaw angles of the target vehicle and each cooperating vehicle's cameras based on their field of view angles, and then sending these angles to the target vehicle and each cooperating vehicle to ensure that the edges of the field of view angles of adjacent vehicles overlap to form a seamless surround view: Ensure that the overlap ratio of the field of view edges of adjacent vehicles is 20% ± 3%.

5. The method according to claim 3, characterized in that, The steps of acquiring video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and performing three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file include: Acquire video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle; Construct an initial graph optimization model using pose data; Sparse point clouds are recovered from video streams using structure-reconstruction-motion technique; Lightweight neural radiation field technology is used for interpolation of blind areas; An 8K×4K 360° 3D panoramic video file is generated using a surface reconstruction algorithm.

6. The method according to claim 2, characterized in that, In the step of calculating the hash of the 3D panoramic video file using a hash function, writing it to the blockchain platform, and returning the IPFS content identifier to the target vehicle and authorized user terminal: The SHA-256 hash of the 3D panoramic video file is calculated, and the smart contract is called to write it to the Ethereum sidechain, returning the IPFS content identifier to the target vehicle and authorized user terminal.

7. The method according to claim 1, characterized in that, The location coordinates are GNSS coordinates; The video streams captured by the cameras of the target vehicle and each cooperating vehicle are captured frame-level synchronously by the cameras of the target vehicle and each cooperating vehicle through NTP time synchronization, and are uploaded to the cloud through a 5G uRLLC link after being encoded using the H.265 standard.

8. A multi-vehicle collaborative panoramic video recording system based on vehicle network cloud, characterized in that, The system, configured to implement the method as described in any one of claims 1 to 7, comprises: The triggering module is used to respond to the panoramic recording session request initiated by the target vehicle and obtain the target vehicle's position coordinates and expected shooting radius; The collaborative vehicle screening module is used to set a search radius according to the desired shooting radius, and to retrieve several vehicles that are online and whose cameras are idle within the search radius centered on the location coordinates as collaborative vehicles. The adaptive viewing angle adjustment module is used to determine the pitch angle and yaw angle of the target vehicle and each cooperative vehicle's camera based on the field of view angle of the target vehicle and each cooperative vehicle's camera, and send them to the target vehicle and each cooperative vehicle to ensure that the edges of the field of view angles of adjacent vehicles overlap to form a seamless surround view. The spatiotemporal synchronization acquisition module is used to acquire video streams and pose data captured by cameras of the target vehicle and each cooperating vehicle, and to perform three-dimensional fusion of the video streams and pose data to obtain a 360° three-dimensional panoramic video file.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images