A vehicle-mounted perception system and method based on AR glasses

Abstract

The embodiment of the present invention discloses a vehicle-mounted perception system and method based on AR glasses. The system includes: an object recognition module configured to use AR glasses to identify and match the current vehicle's surrounding environment, and determine the identity information and longitude and latitude information of the object to be identified; an information acquisition module configured to obtain other vehicle information and roadside information, and send the other vehicle information and roadside information to a data fusion processing module; the data fusion processing module is configured to fuse other vehicle information, roadside information and the identity information of the object to be identified to obtain a fusion processing result; the information transmission module is configured to send the fusion processing result to the background through a cellular network and to other vehicles through direct communication. Through the technical solution of the embodiment of the present invention, object recognition is performed through AR glasses, and the roadside obstacle information from the driver's perspective is supplemented, thereby improving the driving safety of the vehicle.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of power system image processing, and in particular to a vehicle-mounted perception system and method based on AR glasses. Background Art

[0002] The intelligent transportation system includes a variety of intelligent devices. The intelligence of vehicles involves single-vehicle intelligence and V2X collaborative intelligence. V2X technology obtains real-time road conditions and road information of the traffic environment through communication between vehicles, vehicles and people, and vehicles and roads, thereby improving driving safety. In previous vehicle-road collaborative systems, detection of beyond-visual-range and vulnerable traffic participants is usually performed through roadside cameras and radars, but the detection of fixed infrastructure equipment has certain blind spots, and the reliability of the V2X warning information provided is limited.

[0003] Therefore, how to improve the safety of vehicle driving is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] The embodiment of the present invention provides an in-vehicle perception system and method based on AR glasses, so as to perform object recognition through AR glasses, supplement the roadside obstacle information from the driver's perspective, and improve the driving safety of the vehicle.

[0005] In a first aspect, an embodiment of the present invention provides a vehicle-mounted perception system based on AR glasses, the system comprising:

[0006] The object recognition module is configured to use AR glasses to identify and match the current vehicle's surrounding environment to determine the identity information and longitude and latitude information of the object to be identified; wherein the object to be identified includes obstacles, traffic signs and traffic participants around the current vehicle;

[0007] The information acquisition module is configured to acquire other vehicle information and roadside information, and send the other vehicle information and roadside information to the data fusion processing module;

[0008] The data fusion processing module is configured to fuse other vehicle information, roadside information and identity information of the object to be identified to obtain a fusion processing result;

[0009] The information transmission module is configured to send the fusion processing results to the background through the cellular network and to other vehicles through direct communication.

[0010] In a second aspect, an embodiment of the present invention further provides a vehicle-mounted perception method based on AR glasses, comprising:

[0011] Using AR glasses to identify and match the current vehicle's surroundings, determine the identity information and longitude and latitude information of the objects to be identified; wherein the objects to be identified include obstacles, traffic signs, and traffic participants around the current vehicle;

[0012] Acquire other vehicle information and roadside information, and send the other vehicle information and roadside information to the data fusion processing module;

[0013] Fusing other vehicle information, roadside information, and identity information of the object to be identified to obtain a fusion processing result;

[0014] The fusion processing results are sent to the background through the cellular network and to other vehicles through direct communication.

[0015] The embodiment of the present invention discloses a vehicle-mounted perception system and method based on AR glasses. The system includes: an object recognition module configured to use AR glasses to identify and match the current vehicle surroundings, and determine the identity information and latitude and longitude information of the object to be identified; an information acquisition module configured to obtain other vehicle information and roadside information, and send the other vehicle information and roadside information to a data fusion processing module; a data fusion processing module configured to fuse other vehicle information, roadside information and the identity information of the object to be identified to obtain a fusion processing result; an information transmission module configured to send the fusion processing result to the background through a cellular network and to other vehicles through direct communication. Through the technical solution of the embodiment of the present invention, the car uses AR glasses to identify obstacles and traffic signs on the road during driving, sends the information to the vehicle-side terminal device, generates V2X warning information through data fusion processing, and improves the driving safety of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating preferred embodiments and are not to be considered as limiting the present invention. Also, the same reference symbols are used throughout the drawings to represent the same parts. In the drawings:

[0017] Figure 1 is a structural schematic diagram of an in-vehicle perception system based on AR glasses provided in an embodiment of the present invention;

[0018] Figure 2 is a structural schematic diagram of another vehicle-mounted perception system based on AR glasses provided in an embodiment of the present invention;

[0019] Figure 3This is a schematic diagram of the connection between an AR glasses control host and a vehicle-mounted communication terminal provided in an embodiment of the present invention;

[0020] Figure 4 It is a schematic diagram of a process of AR glasses recognizing road condition information, bus stop signs and station information provided by an embodiment of the present invention;

[0021] Figure 5 It is a schematic diagram of a data fusion processing structure provided by an embodiment of the present invention;

[0022] Figure 6 It is a flow chart of an in-vehicle perception method based on AR glasses provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations (or steps) therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0025] In the past two years, as the Chinese government has vigorously encouraged the development of high-tech industries, augmented reality (AR) technology has become known to most people. Now, with the continuous breakthroughs in technology, AR glasses will gradually be applied to future industrial fields. However, in the automotive field, AR still has a lot of room for development, and no company on the market has applied AR glasses to vehicles. On the one hand, in-vehicle AR applications can provide drivers with better traffic environment information prompts; on the other hand, in the in-vehicle environment, through the connection with the in-vehicle communication terminal, it can bring more entertainment driving experience to non-drivers. Figure 1 is a structural diagram of an in-vehicle perception system based on AR glasses provided in an embodiment of the present invention, see Figure 1The main function of the present invention is that in a vehicle-mounted environment, the driver can wear AR glasses to increase the recognition of pedestrians, vehicles and road test information in the surrounding environment, and obtain road condition information to the AR glasses. The AR glasses communicate with the vehicle-mounted communication terminal to send the corresponding traffic information to the vehicle end, and integrate it with the information of the vehicle-mounted communication terminal. The identified identification content matching location information can be sent to the vehicle-mounted communication terminal VBOX, converted into V2X information, sent to the cloud background and broadcast to other vehicles, further increasing the safety and reliability of vehicle driving.

[0026] Figure 2 is a schematic diagram of the structure of another vehicle-mounted perception system based on AR glasses provided in an embodiment of the present invention. This embodiment can be applied to the situation where AR glasses are used to perceive and obtain the current vehicle roadside information, such as Figure 2 As shown, the vehicle-mounted perception system based on AR glasses provided in an embodiment of the present invention may include: an object recognition module 210, an information acquisition module 220, a data fusion processing module 230 and an information transmission module 240; wherein:

[0027] The object recognition module 210 is configured to use AR glasses to identify and match the current vehicle's surrounding environment to determine the identity information and longitude and latitude information of the object to be identified; wherein the object to be identified includes obstacles, traffic signs and traffic participants around the current vehicle.

[0028] Among them, augmented reality (AR) can refer to a technology that calculates the position and angle of a camera image in real time and adds a corresponding image. The goal of this technology is to overlay the virtual world on the real world on the screen and interact with it. For example, AR glasses determine the user's state by tracking the eye trajectory and display the road information or surrounding buildings. For example, AR glasses are used to track the user's eye trajectory and identify and match the current vehicle's surrounding environment based on feature points.

[0029] The object to be identified may refer to an object to be identified in the current vehicle's surrounding environment, including but not limited to obstacles, traffic signs, and traffic participants around the current vehicle. When detecting an object to be identified, the feature point information of the object to be identified needs to be entered into a library. When the AR glasses scan the object, the feature point information of the object to be identified is matched with the feature point information of the object to be identified in the library to identify the object to be identified.

[0030] Identity information may refer to information that can characterize the attributes of an object, including but not limited to the state, category, location, and name of the object; for example, AR glasses are used to identify and match the current vehicle surroundings to determine the name of the object to be identified.

[0031] The information acquisition module 220 is configured to acquire other vehicle information and roadside information, and send the other vehicle information and roadside information to the data fusion processing module.

[0032] Among them, other vehicle information may refer to other vehicle information except the current vehicle, including but not limited to the motion state, speed, position and size of other vehicles.

[0033] Roadside information may refer to roadside information obtained by roadside cameras, radars and other equipment; for example, roadside information obtained by roadside cameras, radars and other equipment includes but is not limited to the road information on which the current vehicle is located and roadside obstacle information.

[0034] Optionally, the information acquisition module includes:

[0035] Using an information acquisition device to acquire roadside information and sending the roadside information to a data fusion processing module; wherein the information acquisition device includes a radar and a camera; the roadside information includes current vehicle driving road environment information;

[0036] The current motion state information of other vehicles is obtained; the motion state information is transmitted through direct communication between vehicles.

[0037] The communication terminal VBOX includes but is not limited to an information acquisition module and a data fusion processing module. The communication terminal VBOX information receiving obtains information wirelessly. The application layer protocol followed is the HTTP protocol and the transport layer application protocol uses the UDP protocol. The wireless transmission Wi-Fi version is 802.11ac 2x2. Figure 3 This is a schematic diagram of the connection between an AR glasses control host and a vehicle-mounted communication terminal provided in an embodiment of the present invention, see Figure 3 , the AR glasses host and the vehicle-side communication terminal VBOX perform wireless transmission through the wifi interface, and the AR glasses control host transmits the identity information and latitude and longitude information of the object to be identified to VBOX through wireless. The UDP protocol is selected in the embodiment of the present invention because the UDP protocol is a connectionless protocol, that is, it does not need to establish a connection in advance before transmitting data. The UDP protocol is simpler than the TCP protocol.

[0038] The data fusion processing module 230 is configured to fuse other vehicle information, roadside information and identity information of the object to be identified to obtain a fusion processing result.

[0039] Among them, fusion processing can refer to the intelligent synthesis of various information through processing to produce more accurate, complete and reliable estimates and judgments than single information; for example, other vehicle information, roadside information and identity information of the object to be identified are processed to determine the location information and size of the object to be identified.

[0040] The information transmission module 240 is configured to send the fusion processing results to the background through the cellular network and to other vehicles through direct communication.

[0041] Among them, the cellular network may refer to a mobile communication hardware architecture, and the cellular network includes but is not limited to GSM network, CDMA network, 3G network, FDMA, TDMA, 4G network and 5G network.

[0042] Direct communication may refer to the roadside device, the vehicle-mounted device and the portable radio device realizing direct communication and information exchange between vehicles, between vehicles and roads, and between vehicles and people through radio transmission. For example, the vehicle-mounted device sends the fusion processing result to other vehicles through radio transmission.

[0043] Optionally, the information transmission module includes:

[0044] Upload the fusion processing results to the cloud backend in the V2X message format through the cellular network, and broadcast the fusion processing results to the intelligent transportation system through the cloud backend;

[0045] The fusion processing results are sent to other vehicles in the V2X message format through direct vehicle-to-vehicle communication.

[0046] Among them, the communication terminal virtual machine (Virtual Box, VBOX) sends the fusion processing results to the background through the cellular network and to other vehicles through direct communication. The data is sent to the background through the cellular network, and the fusion processing results are uploaded to the cloud background in the format of vehicle to everything (Vihicle to everything, V2X) through the message transmission module and the message processing module, and the data is broadcast to the intelligent transportation system through the cloud background. The fusion processing results are sent to the remote vehicle in the V2X format through the direct communication between vehicles, and the information is sent to other vehicles through VBOX.

[0047] The embodiment of the present invention provides a vehicle-mounted perception system based on AR glasses, the system comprising: an object recognition module configured to use AR glasses to identify and match the current vehicle surrounding environment, determine the identity information and latitude and longitude information of the object to be identified; an information acquisition module configured to obtain other vehicle information and roadside information, and send the other vehicle information and roadside information to a data fusion processing module; the data fusion processing module configured to fuse other vehicle information, roadside information and the identity information of the object to be identified to obtain a fusion processing result; an information transmission module configured to send the fusion processing result to the background through a cellular network and to other vehicles through direct communication. Through the technical solution of the embodiment of the present invention, the current vehicle uses AR glasses to identify objects to be identified and traffic signs on the road during driving. When detecting an object to be identified, the feature point information of the object to be identified needs to be entered into a library. When the AR glasses scan the object, it matches the feature point information of the object to be identified in the library to identify the object to be identified, and sends the information to the vehicle-side terminal device. Through data fusion processing, V2X warning information is generated to improve the driving safety of the vehicle.

[0048] The embodiment of the present invention further optimizes the above embodiment on the basis of the above embodiment, and the embodiment of the present invention can be combined with each optional solution in one or more of the above embodiments. In the vehicle-mounted perception system based on AR glasses provided in the embodiment of the present invention, the object recognition module includes:

[0049] Scan the current vehicle's surroundings through AR glasses to obtain feature point information of the object to be identified; wherein the feature point information includes the shape and size of the object to be identified;

[0050] Match the feature point information of the object to be identified with the feature point information of the identified object in the information database to determine the identity information of the object to be identified;

[0051] The identity information of the object to be identified is sent to the data fusion processing module.

[0052] Among them, the hardware and software required for AR glasses object recognition are as follows:

[0053] The hardware is the camera of the AR glasses. Table 1 is a camera composition table of the AR glasses provided in an embodiment of the present invention. Referring to Table 1, the current vehicle surroundings are scanned by the AR glasses to obtain feature point information of the object to be identified.

[0054] Table 1 Camera composition of AR glasses

[0055] Head Tracking 4 visible light cameras, Bluetooth Eye Tracking 2 infrared (IR) cameras depth 1-MP Time-of-Flight Depth Sensor Inertial Measurement Unit (IMU) Accelerometer, gyroscope, magnetometer camera 8-MP still images, 1080p30 video

[0056] The software includes:

[0057] Unity3D can refer to an AR development engine. AR glasses object recognition is debugged using Unity3D and released to Hololens after successful debugging.

[0058] Visual Studio 2019 may refer to using Visual Studio 2019 to write scripts and publish projects in this development;

[0059] Vuforia may refer to an AR SDK that implements multiple AR features, including but not limited to object recognition.

[0060] Optionally, the object recognition module further includes:

[0061] If the feature points of the object to be identified are matched successfully, the identity information of the object to be identified is retained and sent to the data fusion processing module;

[0062] If the feature points of the object to be identified are not matched successfully, the preset unidentified information is sent to the data fusion processing module.

[0063] Figure 4 This is a schematic diagram of a process for AR glasses to recognize road condition information, bus stop signs and station information provided by an embodiment of the present invention. Figure 4 , Vuforia function can enable AR glasses to have the function of identifying objects, and Vuforia application can run the configuration in Unity on AR glasses. First, after sending the information model of the object to the vuforia background, vuforia will store the information of the object in the database. At the same time, after importing the vuforia package into the AR glasses, vuforia enables AR glasses to have object recognition function. Start extracting feature points. When the information of the object is identified, the information of the identified object will be retained and transmitted to the data processing module. If it cannot be identified, the preset unrecognized information will be transmitted to the data processing module. In this way, the object recognition function is realized; wherein, the recognition includes but is not limited to the recognition of roadside obstacles, the recognition of vehicles in front, and the recognition of roadside pedestrians.

[0064] The embodiment of the present invention provides a vehicle-mounted perception system based on AR glasses, which scans the surrounding environment of the current vehicle through AR glasses to obtain feature point information of the object to be identified; wherein the feature point information includes the shape and size of the object to be identified; the feature point information of the object to be identified is matched with the feature point information of the identified object in the information library to determine the identity information of the object to be identified; and the identity information of the object to be identified is sent to the data fusion processing module. Using the technical solution of the embodiment of the present invention, the current vehicle uses AR glasses to identify objects to be identified on the road and traffic signs during driving. When the object to be identified is to be detected, the feature point information of the object to be identified needs to be entered into the library. When the AR glasses scan the object, it matches the feature point information of the object to be identified in the library to identify the object to be identified. Through the visual supplement of AR glasses, the recognition range and credibility of the surrounding driving environment can be increased according to the driver's perspective and line of sight, the reliability of V2X warning is improved, and the safety and reliability of vehicle driving are further increased.

[0065] The embodiment of the present invention further optimizes the above embodiment on the basis of the above embodiment, and the embodiment of the present invention can be combined with each optional solution in one or more of the above embodiments. In the vehicle-mounted perception system based on AR glasses provided in the embodiment of the present invention, the data fusion processing module includes:

[0066] The identity information of the object to be identified, other vehicle information and roadside information received at the same time are selected for fusion processing;

[0067] The fusion processing result is obtained and sent to the information transmission module; wherein the fusion processing result includes the position information, size and motion state of the object to be identified.

[0068] Among them, the identity information of the object to be identified, other vehicle information and roadside information received at the same time are selected for fusion processing; the object to be identified obtained by the AR glasses is made to correspond to the current vehicle driving traffic environment according to the database content and the location information provided by the map, so as to display the object to be identified in the display device; wherein, the display device includes but is not limited to mobile phone navigation and vehicle navigation, etc.; through the visual supplement of AR glasses, the recognition range and credibility of the surrounding driving environment can be increased according to the driver's perspective and line of sight. The location information, size and motion state of the object to be identified after fusion processing are sent to the information transmission module, which improves the reliability of V2X warning and further increases the safety and reliability of vehicle driving.

[0069] Optionally, the data fusion processing module further includes:

[0070] Select the identity information of the object to be identified, other vehicle information and roadside information received at the same time;

[0071] Determine the center coordinates of the object to be identified in the world coordinate system based on the latitude and longitude information of the object to be identified;

[0072] Sorting the objects to be identified according to the confidence of the objects to be identified that are contained in the circumscribed rectangular frame; the sorting includes from high confidence to low confidence;

[0073] Performing non-maximum suppression processing on at least two adjacent objects to be identified with high confidence, and obtaining the processed objects to be identified;

[0074] Fusion processing is performed based on the processed object to be identified, the identity information of the object to be identified, other vehicle information and roadside information to determine a fusion processing result; wherein the fusion processing result includes the position information and size of the object to be identified.

[0075] Among them, the vehicle-side communication terminal VBOX will integrate and process the information received from AR glasses, other vehicles and the roadside, identify the location information and size of obstacles, and conduct V2X warning broadcasts.

[0076] The data information fusion processing process is as follows:

[0077] A1: Decode the road side message (RSM) from the AR glasses, vehicle, and roadside, and select the ones with the same timestamp to fuse.

[0078] A2: Calculate the coordinates of the center point of the object to be identified in the world coordinate system according to the longitude and latitude of the object to be identified;

[0079] A3: Sort all the objects to be identified in each bounding rectangle containing the object to be identified from large to small according to their confidence level;

[0080] A4: Calculate the coordinates of the corner points of the circumscribed rectangular frame of two adjacent objects to be identified;

[0081] A5: Perform non-maximum suppression (NMS) calculation based on the corner coordinates of the two objects to be identified;

[0082] A6: Provides the result after all the bounding rectangles are redundant.

[0083] in, Figure 5 is a schematic diagram of a data fusion processing structure provided by an embodiment of the present invention, see Figure 5, send the information obtained by the AR glasses, the roadside information and the remote vehicle information to the data fusion processing module, put the data to be fused and processed into the message queue of the road test unit, and regularly detect the data in the message queue within a preset time, determine whether the information obtained by the AR glasses, the roadside information and the remote vehicle information are in the same time period, determine the center point coordinates of the object to be identified in the world coordinate system, perform non-maximum suppression processing on at least two adjacent objects to be identified with high confidence, and obtain the processed object to be identified; perform fusion processing based on the processed object to be identified, the identity information of the object to be identified, other vehicle information and the roadside information, determine the fusion processing result, and obtain the basic safety message (basicsafety message, BSM) of the fusion processing result.

[0084] The embodiment of the present invention provides a vehicle-mounted perception system based on AR glasses, which selects the identity information of the object to be identified, other vehicle information and roadside information received at the same time for fusion processing; obtains the fusion processing result and sends it to the information transmission module; wherein the fusion processing result includes the position information, size and motion state of the object to be identified. Using the technical solution of the embodiment of the present invention, the current vehicle uses AR glasses to identify the object to be identified on the road during driving, and sends the information to the vehicle-side VBOX, by selecting the identity information of the object to be identified, other vehicle information and roadside information received at the same time; determines the center point coordinates of the object to be identified in the world coordinate system; performs non-maximum suppression processing on at least two adjacent objects to be identified with high confidence, and obtains the processed object to be identified; performs fusion processing based on the processed object to be identified, the identity information of the object to be identified, other vehicle information and roadside information, determines the position information and size of the object to be identified in the map according to the database content and the position information provided by the map, generates V2X warning information, and improves the safety and reliability of vehicle driving. The addition of AR glasses recognition source supplements the roadside obstacle information from the driver's perspective and improves the detection range and reliability of V2X warnings.

[0085] Figure 6 is a flow chart of an in-vehicle perception method based on AR glasses provided in an embodiment of the present invention. The embodiment of the present invention can be applied to the situation where AR glasses are used to perceive and obtain the current vehicle roadside information. The method can be executed by the in-vehicle perception system based on AR glasses provided in any embodiment of the present invention. Figure 6 As shown, the vehicle-mounted sensing method based on AR glasses provided in an embodiment of the present invention specifically includes the following steps:

[0086] S610: Use AR glasses to identify and match the current vehicle surrounding environment to determine the identity information and longitude and latitude information of the object to be identified.

[0087] The objects to be identified include obstacles, traffic signs and traffic participants around the current vehicle.

[0088] On the basis of the above embodiment, optionally, the AR glasses are used to identify and match the current vehicle surrounding environment to determine the identity information and latitude and longitude information of the object to be identified, including:

[0089] Scan the current vehicle's surroundings through AR glasses to obtain feature point information of the object to be identified; wherein the feature point information includes the shape and size of the object to be identified;

[0090] Match the feature point information of the object to be identified with the feature point information of the identified object in the information database to determine the identity information of the object to be identified;

[0091] The identity information of the object to be identified is sent to the data fusion processing module.

[0092] Based on the above embodiment, optionally, the using of AR glasses to identify and match the current vehicle surroundings to determine the identity information and latitude and longitude information of the object to be identified also includes:

[0093] If the feature points of the object to be identified are matched successfully, the identity information of the object to be identified is retained and sent to the data fusion processing module;

[0094] If the feature points of the object to be identified are not matched successfully, the preset unidentified information is sent to the data fusion processing module.

[0095] S620: Acquire other vehicle information and roadside information, and send the other vehicle information and roadside information to a data fusion processing module.

[0096] On the basis of the above embodiment, optionally, other vehicle information, roadside information and identity information of the object to be identified are fused to obtain a fusion processing result, including:

[0097] Using an information acquisition device to acquire roadside information and sending the roadside information to a data fusion processing module; wherein the information acquisition device includes a radar and a camera; the roadside information includes current vehicle driving road environment information;

[0098] The current motion state information of other vehicles is obtained; the motion state information is transmitted through direct communication between vehicles.

[0099] S630: Fusing other vehicle information, roadside information, and identity information of the object to be identified to obtain a fusion processing result.

[0100] On the basis of the above embodiment, optionally, the fusion processing of other vehicle information, roadside information and identity information of the object to be identified to obtain the fusion processing result includes:

[0101] The identity information of the object to be identified, other vehicle information and roadside information received at the same time are selected for fusion processing;

[0102] The fusion processing result is obtained and sent to the information transmission module; wherein the fusion processing result includes the position information, size and motion state of the object to be identified.

[0103] On the basis of the above embodiment, optionally, the selecting and fusing the identity information of the object to be identified, other vehicle information and roadside information received at the same time includes:

[0104] Select the identity information of the object to be identified, other vehicle information and roadside information received at the same time;

[0105] Determine the center coordinates of the object to be identified in the world coordinate system based on the latitude and longitude information of the object to be identified;

[0106] Sorting the objects to be identified according to the confidence of the objects to be identified that are contained in the circumscribed rectangular frame; the sorting includes from high confidence to low confidence;

[0107] Performing non-maximum suppression processing on at least two adjacent objects to be identified with high confidence, and obtaining the processed objects to be identified;

[0108] Fusion processing is performed based on the processed object to be identified, the identity information of the object to be identified, other vehicle information and roadside information to determine a fusion processing result; wherein the fusion processing result includes the position information and size of the object to be identified.

[0109] S640: Send the fusion processing result to the background through the cellular network and to other vehicles through direct communication.

[0110] Based on the above embodiment, optionally, sending the fusion processing result to the background through the cellular network and to other vehicles through direct communication includes:

[0111] Upload the fusion processing results to the cloud backend in the V2X message format through the cellular network, and broadcast the fusion processing results to the intelligent transportation system through the cloud backend;

[0112] The fusion processing results are sent to other vehicles in the V2X message format through direct vehicle-to-vehicle communication.

[0113] The vehicle-mounted sensing method based on AR glasses provided in the embodiments of the present invention can be applied to the vehicle-mounted sensing system based on AR glasses provided in any of the above-mentioned embodiments of the present invention, and has the corresponding functions and beneficial effects of the vehicle-mounted sensing system based on AR glasses. For technical details not described in detail in the above-mentioned embodiments, please refer to the vehicle-mounted sensing system based on AR glasses provided in any of the embodiments of the present application.

[0114] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0115] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A vehicle-mounted perception system based on AR glasses, characterized in that: The system comprises: The object recognition module is configured to use AR glasses to identify and match the current vehicle's surrounding environment to determine the identity information and latitude and longitude information of the object to be identified; wherein the object to be identified includes obstacles, traffic signs and traffic participants around the current vehicle; The information acquisition module is configured to acquire other vehicle information and roadside information, and send the other vehicle information and roadside information to the data fusion processing module; The data fusion processing module is configured to fuse other vehicle information, roadside information and identity information of the object to be identified to obtain a fusion processing result; The information transmission module is configured to send the fusion processing results to the backend via the cellular network and to other vehicles via direct communication; Wherein, the data fusion processing module also includes: Select the identity information, other vehicle information and roadside information of the object to be identified received at the same time; determine the center point coordinates of the object to be identified in the world coordinate system according to the latitude and longitude information of the object to be identified; sort the objects to be identified according to the confidence of the objects to be identified that contain the circumscribed rectangular frame; the sorting includes from high confidence to low confidence; perform non-maximum suppression processing on at least two adjacent objects to be identified with high confidence to obtain the processed objects to be identified; perform fusion processing based on the processed objects to be identified, the identity information of the objects to be identified, other vehicle information and roadside information, and determine the fusion processing result; wherein the fusion processing result includes the position information and size of the objects to be identified; The object recognition module also includes: The current vehicle surroundings are scanned by AR glasses to obtain feature point information of the object to be identified; wherein the feature point information includes the shape and size of the object to be identified; the feature point information of the object to be identified is matched with the feature point information of the identified object in the information library to determine the identity information of the object to be identified; the identity information of the object to be identified is sent to the data fusion processing module; wherein the AR glasses have an object recognition function.

2. The system according to claim 1, characterized in that The object recognition module also includes: If the feature points of the object to be identified are matched successfully, the identity information of the object to be identified is retained and sent to the data fusion processing module; If the feature points of the object to be identified are not matched successfully, the preset unidentified information is sent to the data fusion processing module.

3. The system according to claim 1, characterized in that The information acquisition module includes: Using an information acquisition device to acquire roadside information and sending the roadside information to a data fusion processing module; wherein the information acquisition device includes a radar and a camera; the roadside information includes current vehicle driving road environment information; The current motion state information of other vehicles is obtained; the motion state information is transmitted through direct communication between vehicles.

4. The system according to claim 1, characterized in that The data fusion processing module includes: The identity information of the object to be identified, other vehicle information and roadside information received at the same time are selected for fusion processing; The fusion processing result is obtained and sent to the information transmission module; wherein the fusion processing result includes the position information, size and motion state of the object to be identified.

5. The system according to claim 1, characterized in that The information transmission module comprises: Upload the fusion processing results to the cloud backend in the V2X message format through the cellular network, and broadcast the fusion processing results to the intelligent transportation system through the cloud backend; The fusion processing results are sent to other vehicles in the V2X message format through direct vehicle-to-vehicle communication.

6. A vehicle-mounted sensing method based on AR glasses, characterized in that: The method comprises: The AR glasses are used to identify and match the current vehicle's surrounding environment to determine the identity information and longitude and latitude information of the objects to be identified; wherein the objects to be identified include obstacles, traffic signs, and traffic participants around the current vehicle; Acquire other vehicle information and roadside information, and send the other vehicle information and roadside information to the data fusion processing module; Fusing other vehicle information, roadside information, and identity information of the object to be identified to obtain a fusion processing result; The fusion processing results are sent to the backend via the cellular network and to other vehicles via direct communication; The selecting and fusing the identity information of the object to be identified, other vehicle information and roadside information received at the same time includes: Select the identity information, other vehicle information and roadside information of the object to be identified received at the same time; determine the center point coordinates of the object to be identified in the world coordinate system according to the latitude and longitude information of the object to be identified; sort the objects to be identified according to the confidence of the objects to be identified that contain the circumscribed rectangular frame; the sorting includes from high confidence to low confidence; perform non-maximum suppression processing on at least two adjacent objects to be identified with high confidence to obtain the processed objects to be identified; perform fusion processing based on the processed objects to be identified, the identity information of the objects to be identified, other vehicle information and roadside information, and determine the fusion processing result; wherein the fusion processing result includes the position information and size of the objects to be identified; The AR glasses are used to identify and match the current vehicle surrounding environment to determine the identity information and latitude and longitude information of the object to be identified, including: The current vehicle surroundings are scanned by AR glasses to obtain feature point information of the object to be identified; wherein the feature point information includes the shape and size of the object to be identified; the feature point information of the object to be identified is matched with the feature point information of the identified object in the information library to determine the identity information of the object to be identified; the identity information of the object to be identified is sent to the data fusion processing module; wherein the AR glasses have an object recognition function.

7. The method according to claim 6, characterized in that The AR glasses are used to identify and match the current vehicle surrounding environment to determine the identity information and latitude and longitude information of the object to be identified, and further include: If the feature points of the object to be identified are matched successfully, the identity information of the object to be identified is retained and sent to the data fusion processing module; If the feature points of the object to be identified are not matched successfully, the preset unidentified information is sent to the data fusion processing module.

Images