Message fusion method and device, storage medium and electronic device

Abstract

The invention discloses a message fusion method and device, a storage medium and an electronic device, and relates to the field of traffic communication.The message fusion method comprises the steps that under the condition that M messages are obtained through N message channels and at least one message is obtained through each message channel in the N message channels, N messages are selected from the M messages, wherein the message channels corresponding to the N messages are different from one another, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N; and carrying out fusion processing on the N messages to obtain a fused message. By adopting the technical scheme, the problem of low message transmission efficiency is solved.

Description

Technical Field

[0001] The present application relates to the field of transportation communications, and specifically, to a message fusion method and device, a storage medium, and an electronic device. Background Art

[0002] As the main means of transportation in the low-altitude economy, flying cars have the dual functions of aircraft and cars, involving the transmission of multiple different source messages. These messages are usually generated at different frequencies. If they cannot be accurately fused, the accuracy of subsequent algorithms will be affected, thus affecting the safety of the flight. Therefore, how to achieve accurate fusion of multi-source data in a complex environment has become an important technical problem in the development of flying car communication middleware.

[0003] At present, flying vehicles generally use the Controller Area Network (CAN) bus message transmission method because they do not involve multiple sensors, especially cameras and LiDAR (Light Detection and Ranging). This transmission method is obviously not suitable for flying cars. The Robot Operating System (ROS), a middleware used for robot communication, can only receive messages from one topic at a time and cannot receive messages from multiple topics at the same time, resulting in low message transmission efficiency.

[0004] With respect to the problem of low efficiency of message transmission in the related art, no effective solution has been proposed so far. Therefore, it is necessary to improve the related art to overcome the above-mentioned defects in the related art. Summary of the invention

[0005] The embodiments of the present application provide a message fusion method and device, a storage medium, and an electronic device to at least solve the problem of low efficiency of message transmission.

[0006] According to one aspect of an embodiment of the present application, a message fusion method is provided, comprising: when M messages are obtained through N message channels and at least one message is obtained through each of the N message channels, selecting N messages from the M messages, wherein the message channels corresponding to the N messages are different, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N; fusing the N messages to obtain a fused message.

[0007] In an exemplary embodiment, selecting N messages from the M messages includes: determining the timestamp of each message in the M messages; selecting a designated message from the messages corresponding to a preset message channel in the N message channels, wherein the designated message is a message whose timestamp is closest to a preset moment in the messages corresponding to the preset message channel, and the preset moment is a moment for determining to fuse the N messages; selecting N-1 messages from the messages corresponding to N-1 message channels, wherein the j-th message among the N-1 messages is a message whose timestamp is closest to the preset moment in the messages corresponding to the j-th message channel in the N-1 message channels, and whose timestamp is within a preset time length after the timestamp of the designated message, and the N-1 message channels are message channels among the N message channels excluding the preset message channel, and j is a positive integer less than or equal to N-1; wherein the N messages include the designated message and the N-1 messages.

[0008] In an exemplary embodiment, selecting N messages from the M messages includes: obtaining a message identifier of each of the M messages; and selecting N messages from the M messages, wherein the N messages have the same message identifier.

[0009] In an exemplary embodiment, selecting N messages from the M messages includes: when the message identifier of any message in the messages corresponding to the target message channel in the N message channels is a reference identifier, querying the message identifier of the message corresponding to each message channel in the N-1 message channels, wherein the N-1 message channels are message channels in the N message channels other than the target message channel, the message identifier includes the reference identifier, and the target message channel is any message channel in the N message channels; when the message corresponding to each message channel in the N-1 message channels has a message with the reference identifier, selecting a message with the reference identifier from the messages corresponding to each message channel in the N message channels to obtain the N messages.

[0010] In an exemplary embodiment, the method further includes: when there are no N target messages in the messages corresponding to the N message channels, continuing to obtain messages through the N message channels, wherein the message channels corresponding to the N target messages are different from each other and have the same message identifier.

[0011] In an exemplary embodiment, the N messages are merged to obtain a merged message, including: sorting the N messages in the order of their corresponding timestamps; or sorting the N messages according to a preset arrangement order of the N message channels; or sorting the N messages according to their corresponding message priorities to obtain N sorted messages; and fusing the sorted N messages in the corresponding sort order to obtain a fused message.

[0012] In an exemplary embodiment, the N sorted messages are merged in a corresponding sorting order to obtain a merged message, including: merging the first message and the second message in the N sorted messages to obtain a reference message; looping the following operations, and determining the reference message after the loop is completed as the merged message, wherein, in the first case of performing the following operations, i is equal to 3: when there is an i-th message in the N sorted messages, merging the reference message with the i-th message, determining the merged message as the reference message, and updating the value of i to i+1; when there is no i-th message in the N sorted messages, exiting the loop.

[0013] According to another aspect of an embodiment of the present application, a message fusion device is also provided, including: a selection module, used to select N messages from the M messages when M messages are obtained through N message channels and at least one message is obtained through each of the N message channels, wherein the message channels corresponding to the N messages are different, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N; a fusion module, used to fuse the N messages to obtain a fused message.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein the program is configured to execute the above-mentioned message fusion method when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, wherein the processor is configured to execute the above-mentioned message fusion method through the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the message fusion method.

[0017] Through the present application, the application middleware obtains multiple messages through multiple message channels, and when at least one message is obtained through each message channel, one message is selected from the messages obtained from each message channel, and the selected messages are merged, thereby realizing the message fusion function of the middleware. Since the messages are selected and then merged, the accuracy of message fusion and the efficiency of message transmission are improved, thereby solving the problem of low efficiency of message transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 It is a hardware structure block diagram of a mobile terminal of a message fusion method according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a message fusion method according to an embodiment of the present application;

[0022] Figure 3 This is an example diagram of a message not being merged according to a timestamp according to an embodiment of the present application;

[0023] Figure 4 is an example diagram of another message not being merged according to a timestamp according to an embodiment of the present application;

[0024] Figure 5 This is an example diagram of message fusion according to an embodiment of the present application;

[0025] Figure 6 This is an example diagram of a message not being merged according to an identifier according to an embodiment of the present application;

[0026] Figure 7 is an example diagram of another message fusion according to an embodiment of the present application;

[0027] Figure 8 It is a structural block diagram of a message fusion device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of a message fusion method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (MP) or a programmable logic device (FPGA)) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for detecting pneumatic imbalance in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a message fusion method is provided, and middleware is applied. The middleware is used to perform message fusion. Figure 2 is a flow chart of a message fusion method according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps S202-S204:

[0034] Step S202: when M messages are obtained through N message channels, and at least one message is obtained through each of the N message channels, N messages are selected from the M messages, wherein the message channels corresponding to the N messages are different from each other, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N;

[0035] Alternatively, if Figure 3 As shown, when there are N message channels and one of the message channels does not receive a message, the middleware does not perform message fusion and delivery.

[0036] Optionally, the middleware in the embodiment of the present application is a Cyber ​​Real-Time (CyberRT) communication middleware, which can receive messages from multiple message channels. The CyberRT communication middleware can be applied to flying cars.

[0037] Optionally, the N message channels may receive M messages from different sensors or other components (eg, lidar, camera).

[0038] It should be noted that the message channels corresponding to the selected N messages are different, indicating that the sources of the N messages are different, and may come from multiple different cameras, or from a camera, radar, or temperature sensor, etc.

[0039] Step S204: The N messages are merged to obtain a merged message.

[0040] Optionally, after obtaining the N latest messages, the messages are fused, that is, the data of multiple sensors or components are fused into one message for use by subsequent algorithms or components. The fusion processing method includes but is not limited to: time alignment of data, format conversion or algorithm operation, etc., depending on the fusion purpose and usage scenario.

[0041] Optionally, after the middleware obtains the fused message, the fused message may be sent to the next middle section or node.

[0042] It should be noted that the implementation of the message fusion processing in the above steps significantly improves the efficiency and accuracy of data processing in the flying car system. By fusing multi-source data into a unified message, the time difference and format inconsistency between the data are eliminated, so that subsequent algorithms can be processed based on accurately synchronized data, thereby improving the navigation, control and safety of the flying car.

[0043] In the above steps, the application middleware obtains multiple messages through multiple message channels, and when at least one message is obtained through each message channel, one message is selected from the messages obtained from each message channel, and the selected messages are merged, thereby realizing the message fusion function of the middleware. Since the messages are selected and then merged, the accuracy of message fusion and the efficiency of message transmission are improved, thereby solving the problem of low efficiency of message transmission.

[0044] In an exemplary embodiment, selecting N messages from the M messages may be implemented by following steps S11-S13:

[0045] Step S11: Determine the timestamp of each message in the M messages;

[0046] Optionally, the middleware will accurately record the timestamps of each of the M messages received from the N message channels. The timestamp is a mark of the specific moment when the message is generated, which is crucial for time synchronization.

[0047] Step S12: selecting a designated message from the messages corresponding to the preset message channels in the N message channels, wherein the designated message is a message whose timestamp is closest to a preset time among the messages corresponding to the preset message channels, and the preset time is the time when the N messages are determined to be fused;

[0048] Optionally, the preset message channel (primary message channel) may be determined according to the frequency of receiving messages, or according to the needs of the user, for example Figure 4 Message channel 1 (Channel_1).

[0049] Optionally, the middleware receives the message from a preset message channel (e.g. Figure 4 A message with a timestamp closest to the preset time (i.e., the fusion time) is selected as the designated message (e.g., Figure 4 The message (m2) obtained by Channel_1 in the example.

[0050] It should be noted that the selection of designated messages ensures that the messages in the main message channel are the latest and closest to the fusion time, which helps to improve the timeliness and relevance of the fused messages and ensure that the flying car system can make decisions based on the latest data.

[0051] Step S13: Select N-1 messages from the messages corresponding to N-1 message channels, wherein the j-th message among the N-1 messages is a message whose timestamp is closest to the preset time and whose timestamp is within a preset time length after the timestamp of the designated message among the messages corresponding to the j-th message channel among the N-1 message channels, and the N-1 message channels are message channels among the N message channels except the preset message channel, and j is a positive integer less than or equal to N-1; wherein the N messages include the designated message and the N-1 messages.

[0052] Optionally, the preset duration is 5 minutes.

[0053] Alternatively, if Figure 4 As shown, there are N message channels, and each message channel has obtained at least one message, and one of the message channels has obtained a message (for example Figure 4 The timestamp of the message (m1) of the message channel 3 (Channel_3) is in the specified message ( Figure 4 If the time is outside of five minutes after the timestamp of m2) obtained by Channel_1, the middleware will not merge and send the message.

[0054] Alternatively, if Figure 5As shown, there are three message channels, namely Channel_1, Channel_2, and Channel_3. The preset message channel (main message channel) is Channel_1, and the specified message is m2 of Channel_1. At this time, the timestamps of m1 of Channel_2 and m2 of Channel_3 are both within five minutes after the timestamp of m2 of Channel_1. The middleware performs message fusion, and fuses m2 of Channel_1, m1 of Channel_2, and m2 of Channel_3. After obtaining the fused messages, they are uniformly sent to the next node.

[0055] It should be noted that the preset duration can be a predefined threshold of allowed time differences, which ensures that all messages involved in the fusion are relatively close in time and meet the requirements of time synchronization. Through this selection strategy based on timestamps and preset durations, the middleware can ensure that the messages received on different message channels are basically aligned in time. Even if these data come from sensors of different frequencies, they can be fused to generate unified and time-synchronized message data, thereby avoiding data inconsistency problems caused by time differences and improving the accuracy and safety of the algorithm in the flying car system.

[0056] It should be noted that the implementation of the above steps enables the flying car communication middleware to achieve accurate message fusion based on timestamps, and maintain the time consistency of messages even when processing large amounts of data from sensors with different frequencies. This technical strategy improves the efficiency of data utilization and reduces algorithm errors caused by data asynchrony, which is of great significance for the real-time navigation, control and safe flight of flying cars.

[0057] In an exemplary embodiment, selecting N messages from the M messages may also be implemented by following the steps S21-S22:

[0058] Step S21: Obtain a message identifier of each of the M messages;

[0059] Optionally, the middleware first reads the M messages obtained from the N message channels and extracts the message identification (ID) of each message. The message identification can be a unique identifier used to distinguish different types of messages, which is the basis of the ID fusion strategy and helps to ensure that the fused messages have the same data type or relevance, thereby improving the accuracy and reliability of data fusion.

[0060] Step S22: Select N messages from the M messages, wherein the N messages have the same message identifier.

[0061] Optionally, the middleware selects N messages with the same identifier from M messages as fusion objects. These N messages come from different message channels, but they all contain information with the same identifier, indicating that they are different parts or perspectives of the same data set.

[0062] It should be noted that the ID-based message selection strategy ensures the consistency of the types of messages involved in the fusion and their logical relevance. For example, when processing multi-camera images (such as the Bird's Eye View (BEV)) fusion, all images with the same ID will be regarded as different perspectives of the same moment or scene, so that images can be stitched or analyzed more accurately, improving the environmental perception ability of the flying car and the decision-making accuracy of the autonomous driving.

[0063] In an exemplary embodiment, selecting N messages from the M messages may be implemented by following the steps S31-S32:

[0064] Step S31: when the message identifier of any one of the messages corresponding to the target message channel in the N message channels is a reference identifier, query the message identifier of the message corresponding to each message channel in the N-1 message channels, wherein the N-1 message channels are message channels other than the target message channel in the N message channels, the message identifier includes the reference identifier, and the target message channel is any one of the N message channels;

[0065] Optionally, when any one of the N message channels receives a message with an identifier, the middleware will immediately query whether other message channels in the N message channels have received a message with the same identifier.

[0066] Step S32: When a message with the reference identifier exists in the messages corresponding to each of the N-1 message channels, a message with the reference identifier is selected from the messages corresponding to each of the N message channels to obtain the N messages.

[0067] Alternatively, if Figure 6 As shown, there are three message channels, namely Channel_1, Channel_2, and Channel_3. When Channel_1 receives message m1 with ID 1, it will immediately query the IDs of the messages received by Channel_2 and Channel_3. However, at this time, the ID of message m0 received by Channel_3 is 0, and the middleware does not merge and send the messages.

[0068] Alternatively, if Figure 7As shown, there are three message channels, namely Channel_1, Channel_2, and Channel_3. When Channel_1 receives message m1 with ID 1, it will immediately query the IDs of the messages received by Channel_2 and Channel_3. At this time, the ID of message m1 in Channel_2 is 1 and the ID of message m1 in Channel_3 is 1. The middleware then performs message fusion, fusion of m1 of Channel_1, m1 of Channel_2, and m1 of Channel_3, and sends the fused messages to the next node.

[0069] It should be noted that by ensuring that messages matching the reference identifier exist in all message channels involved in the fusion, this step further enhances the logical consistency and time synchronization of message fusion, ensuring that the fused message set contains the latest data of all relevant sensors or components, thereby improving the accuracy and reliability of decisions based on fused data in the flying car system.

[0070] It should be noted that the above-mentioned fusion strategy based on message identification improves the flexibility and efficiency of the flying car communication middleware in processing multi-source data, and is particularly suitable for advanced autonomous driving functions that require integrated multi-angle or multi-sensor data for decision-making. By setting reference identification and performing cross-channel matching, it is possible to ensure the logical and temporal consistency of all data involved in the fusion, avoid fusion errors caused by data mismatch or time difference, reduce unnecessary data processing delays, and have significant technical effects on the real-time navigation, control and safe flight of flying cars.

[0071] In an exemplary embodiment, the method further includes the following steps: when there are no N target messages in the messages corresponding to the N message channels, continue to obtain messages through the N message channels, wherein the message channels corresponding to the N target messages are different from each other and have the same message identifier.

[0072] Optionally, when the middleware attempts to merge N target messages with the same message identifier (ID), but finds that the target message is missing from the messages obtained through one of the N message channels, the middleware does not immediately perform the fusion process, but continues to obtain messages through the N message channels until the target message with the corresponding message identifier is obtained through each channel.

[0073] It should be noted that by continuously obtaining messages, it is ensured that the fusion process will not be based on incomplete or missing data, thereby avoiding the risk of making decisions based on erroneous or incomplete information, and improving the accuracy of the flying car system's environmental perception, path planning and other tasks based on fused data.

[0074] In an exemplary embodiment, the N messages are merged to obtain a merged message, which can be achieved by the following steps S41-S42:

[0075] Step S41: sorting the N messages according to the order of their corresponding timestamps; or sorting the N messages according to the preset arrangement order of the N message channels; or sorting the N messages according to their corresponding message priorities to obtain sorted N messages;

[0076] Optionally, when N messages are selected based on their respective timestamps, the middleware sorts the N messages according to their respective timestamps; when N messages are selected based on their respective identifiers, the middleware sorts the N messages according to the preset arrangement order of the message channel; in addition, the middleware can also sort the N messages according to the priority of each message in the N messages, and the priority can be set according to the message type, importance or urgency (such as fault reporting messages).

[0077] Step S42: The sorted N messages are merged according to the corresponding sorting order to obtain a merged message.

[0078] Optionally, the fusion process involves splicing of data in the message, time synchronization, format conversion, etc., and finally generates a fused message, which contains key information of N original messages and follows the logical order of fusion.

[0079] It should be noted that fusion processing ensures the temporal and logical continuity and consistency of fused messages, which is crucial for flying cars to perform autonomous driving tasks that require precise time or logical sequence. The fused messages can be used more efficiently by downstream components, reducing the intermediate links in data processing, speeding up the response time of the flying car system from data acquisition to decision execution, and improving flight safety.

[0080] It should be noted that the above steps not only improve the efficiency and accuracy of the flying car communication middleware in processing multi-source data, but also enhance the system's robustness and adaptability to complex environments. By ensuring the time consistency, logical order and priority processing of fused data, flying cars can perceive the surrounding environment more accurately and make faster and safer decisions, which has a significant technical effect on the flying car's autonomous driving technology and safe flight.

[0081] In an exemplary embodiment, the sorted N messages are merged according to the corresponding sorting order to obtain the merged message, which can be achieved by the following steps S51-S52:

[0082] Step S51: merging the first message and the second message of the N sorted messages to obtain a reference message;

[0083] Optionally, when N is equal to 2, the reference message is the fused message.

[0084] It should be noted that the reference message generated by the initial fusion provides a benchmark for subsequent fusion, ensuring that the fusion process can be carried out based on a unified starting point, which is conducive to maintaining the time or logical consistency of the final fusion message.

[0085] Step S52: cyclically execute the following steps S521-S522, and determine the reference message after the cycle as the fused message, wherein, in the case of the first execution of the following steps, i is equal to 3:

[0086] Step S521: when there is an i-th message in the sorted N messages, merge the reference message with the i-th message, determine the merged message as the reference message, and update the value of i to i+1;

[0087] It should be noted that starting from the third message, the middleware will check whether there are more messages that have not been merged. If so, the middleware will merge the current reference message with the next message (the i-th message) to generate a new reference message, update the value of i, and continue the next cycle.

[0088] Step S522: If the i-th message does not exist in the sorted N messages, exit the loop.

[0089] It should be noted that if the i-th message does not exist in the sorted N messages, that is, all messages have been fused, the loop is exited, and the reference message generated by the last iteration is determined to be the final fused message.

[0090] It should be noted that by gradually merging two messages at a time, the errors that may occur when processing a large amount of data at one time can be reduced, the robustness of the fusion processing can be improved, and the algorithm design can be simplified, the consumption of computing resources can be reduced, and the fusion processing speed and real-time performance of the flying car communication middleware can be improved. It also ensures that all messages in the N messages are considered and merged, and the final fusion message generated contains the key information of all the original messages, which improves the integrity of the flying car system data fusion.

[0091] It should be noted that the above steps not only improve the accuracy and efficiency of the message fusion of the flying car communication middleware, but also enhance the system's adaptability to the needs of multi-source data fusion, which is an efficient strategy for data fusion processing in flying car communication technology. This fusion method is particularly suitable for processing sensor data that requires time consistency, logical order and high real-time performance, and provides strong technical support for key tasks such as autonomous driving and environmental perception of flying cars.

[0092] Obviously, the above-described embodiments are only embodiments of a part of the present invention, rather than all embodiments. In order to better understand the above method, the above process is described below in conjunction with embodiments, but it is not intended to limit the technical solutions of the embodiments of the present invention, specifically:

[0093] Overall, this application provides a multi-message fusion method based on CyberRT communication middleware in a flying car, which can simultaneously receive and fuse multiple types of messages during simulation, testing, and operation, and generate structured data in real time for message sending and receiving.

[0094] 1. Based on the main message time synchronization strategy:

[0095] The latest message in the first message channel is used as the master message. When the remaining channels have message data and the time difference between the latest message data and the latest message data of the master is within a specified threshold range, the latest message data of all channels is taken to trigger the callback. This strategy can be applied to point cloud mosaics based on multiple point cloud data and inertial measurement unit (IMU) data.

[0096] For example:

[0097] A CyberRT component receives three types of messages at the same time (there are three channels);

[0098] Figure 3 Channel_1 is the main message channel. At this time, Channel_2 does not receive the message, so it cannot merge the messages and does not send the messages.

[0099] Figure 4 Channel_1 is the main message channel. At this time, the timestamps of the messages m1 of Channel_2 and m2 of Channel_1 are within the threshold range, but the timestamps of the messages m1 of Channel_3 and m2 of Channel_1 are not within the threshold range. Therefore, the messages cannot be merged and are not sent.

[0100] Figure 5 In the example, Channel_1 is the main message channel. At this time, the timestamp of the message m1 of Channel_2 compared with the message m2 of Channel_1 is within the threshold range, and the timestamp of the message m2 of Channel_3 compared with the message m2 of Channel_1 is within the threshold range. Message fusion is performed, and the messages m2 of Channel_1, m1 of Channel_2, and m2 of Channel_3 are fused and sent to the next component node.

[0101] 2. Synchronization strategy based on message ID:

[0102] When any Channel receives a message, the message ID exists in all other Channels. At this time, the data with the same ID is synchronized and the callback is triggered. Otherwise, the callback is not triggered. This strategy can be used if the algorithm needs to receive image data from multiple cameras at the same time (such as the BEV algorithm).

[0103] For example:

[0104] Figure 6 The ID of the message in Channel_3 is different from that of the others, so the message cannot be merged and the message is not sent.

[0105] Figure 7 The ID of the latest message in Channel_3 appears in other channels, so the messages are merged and the messages in the three channels are sent to the downstream.

[0106] It should be noted that the present application also has the following advantages: (1) Good real-time performance: CyberRT communication middleware is introduced on flying cars for the first time to ensure the real-time transmission of image and point cloud data; (2) Multiple fusion strategies: A variety of message fusion strategies are provided, which can be selected for different usage scenarios. On the basis of CyberRT's multi-message reception, a main message-based time synchronization strategy and a message ID-based time synchronization strategy are proposed; (3) It realizes the fusion of multiple messages of a component on a flying car and the simultaneous sending and receiving of multiple messages. This is different from the ROS communication middleware, which requires multiple topics to send and receive multiple messages and cannot perform message fusion, thereby reducing the complexity of the code.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0108] In this embodiment, a message fusion device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0109] Figure 8 : is a structural block diagram of a message fusion device according to an embodiment of the present application, the device comprising:

[0110] A selection module 82 is used to select N messages from the M messages when M messages are obtained through N message channels and at least one message is obtained through each of the N message channels, wherein the message channels corresponding to the N messages are different from each other, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N;

[0111] The fusion module 84 is used to fuse the N messages to obtain a fused message.

[0112] The above-mentioned device applies middleware to obtain multiple messages through multiple message channels, and when at least one message is obtained through each message channel, one message is selected from the messages obtained from each message channel, and the selected messages are merged, thereby realizing the message fusion function of the middleware. Since the messages are selected and then merged, the accuracy of message fusion and the efficiency of message transmission are improved, thereby solving the problem of low efficiency of message transmission.

[0113] In an exemplary embodiment, the selection module 82 is also used to determine the timestamp of each message in the M messages; select a specified message from the messages corresponding to the preset message channel in the N message channels, wherein the specified message is the message whose timestamp is closest to the preset time in the messages corresponding to the preset message channel, and the preset time is the time for determining the time to perform fusion processing on the N messages; select N-1 messages from the messages corresponding to the N-1 message channels, wherein the j-th message in the N-1 messages is the message whose timestamp is closest to the preset time in the messages corresponding to the j-th message channel in the N-1 message channels, and the timestamp is within a preset time length after the timestamp of the specified message, the N-1 message channels are the message channels in the N message channels other than the preset message channel, and j is a positive integer less than or equal to N-1; wherein the N messages include the specified message and the N-1 messages.

[0114] In an exemplary embodiment, the selection module 82 is further configured to obtain a message identifier of each of the M messages; and select N messages from the M messages, wherein the N messages have the same message identifier.

[0115] In an exemplary embodiment, the selection module 82 is also used to query the message identifier of the message corresponding to each message channel in the N-1 message channels, when the message identifier of any message in the messages corresponding to the target message channel in the N message channels is a reference identifier, wherein the N-1 message channels are message channels in the N message channels other than the target message channel, the message identifier includes the reference identifier, and the target message channel is any message channel in the N message channels; when the message corresponding to each message channel in the N-1 message channels contains a message with the reference identifier, select a message with the reference identifier from the messages corresponding to each message channel in the N message channels to obtain the N messages.

[0116] In an exemplary embodiment, the above-mentioned device also includes: an acquisition module, which is used to continue to acquire messages through the N message channels when there are no N target messages in the messages corresponding to the N message channels, wherein the message channels corresponding to the N target messages are different from each other and have the same message identifier.

[0117] In an exemplary embodiment, the fusion module 84 is also used to sort the N messages in the order of their corresponding timestamps; or sort the N messages according to the preset arrangement order of the N message channels; or sort the N messages according to their corresponding message priorities to obtain N sorted messages; and fuse the sorted N messages in the corresponding sort order to obtain a fused message.

[0118] In an exemplary embodiment, the fusion module 84 is also used to fuse the first message and the second message among the N sorted messages to obtain a reference message; loop the following operations and determine the reference message after the loop ends as the fused message, wherein, in the first case of performing the following operations, i is equal to 3: when there is an i-th message among the N sorted messages, fuse the reference message with the i-th message, determine the fused message as the reference message, and update the value of i to i+1; when there is no i-th message among the N sorted messages, exit the loop.

[0119] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0120] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0121] S1, when M messages are obtained through N message channels, and at least one message is obtained through each of the N message channels, select N messages from the M messages, wherein the message channels corresponding to the N messages are different, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N;

[0122] S2, fusing the N messages to obtain a fused message.

[0123] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0124] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0125] An embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are performed.

[0126] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0127] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0128] S1, when M messages are obtained through N message channels, and at least one message is obtained through each of the N message channels, select N messages from the M messages, wherein the message channels corresponding to the N messages are different, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N;

[0129] S2, fusing the N messages to obtain a fused message.

[0130] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0131] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0132] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0133] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A message fusion method, characterized in that: Application middleware, which is used for message fusion, includes: In the case where M messages are obtained through N message channels, and at least one message is obtained through each of the N message channels, N messages are selected from the M messages, wherein the message channels corresponding to the N messages are different, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N; The N messages are fused to obtain a fused message.

2. The message fusion method according to claim 1, characterized in that: Selecting N messages from the M messages includes: Determining a timestamp for each of the M messages; Selecting a designated message from the messages corresponding to a preset message channel in the N message channels, wherein the designated message is a message whose timestamp is closest to a preset time among the messages corresponding to the preset message channel, and the preset time is the time when the N messages are determined to be fused; Select N-1 messages from the messages corresponding to N-1 message channels, wherein the j-th message among the N-1 messages is a message whose timestamp is closest to the preset time and whose timestamp is within a preset time length after the timestamp of the specified message among the messages corresponding to the j-th message channel among the N-1 message channels, and the N-1 message channels are message channels among the N message channels except the preset message channel, and j is a positive integer less than or equal to N-1; The N messages include the designated message and the N-1 messages.

3. The method according to claim 1, characterized in that: Selecting N messages from the M messages includes: Obtaining a message identifier of each of the M messages; N messages are selected from the M messages, wherein the N messages have the same message identifier.

4. The method according to claim 3, characterized in that Selecting N messages from the M messages includes: In the case where the message identifier of any one of the messages corresponding to the target message channel in the N message channels is a reference identifier, querying the message identifier of each message channel in the N-1 message channels, wherein the N-1 message channels are message channels other than the target message channel in the N message channels, the message identifiers include the reference identifier, and the target message channel is any one of the N message channels; In the case that the message corresponding to each message channel in the N-1 message channels contains a message with the reference identifier, a message with the reference identifier is selected from the messages corresponding to each message channel in the N message channels to obtain the N messages.

5. The method according to claim 3, characterized in that: The method further comprises: When there are no N target messages among the messages corresponding to the N message channels, continue to obtain messages through the N message channels, wherein the message channels corresponding to the N target messages are different from each other and have the same message identifier.

6. The method according to claim 1, characterized in that The N messages are merged to obtain a merged message, including: Sorting the N messages according to the order of their corresponding timestamps; or sorting the N messages according to the preset arrangement order of the N message channels; or sorting the N messages according to their corresponding message priorities to obtain sorted N messages; The sorted N messages are merged according to the corresponding sorting order to obtain a merged message.

7. The method according to claim 6, characterized in that The sorted N messages are merged according to the corresponding sorting order to obtain a merged message, including: Merging the first message and the second message of the N sorted messages to obtain a reference message; The following operations are performed in a loop, and the reference message after the loop is completed is determined as the fused message, wherein i is equal to 3 when the following operations are performed for the first time: In the case that the i-th message exists in the sorted N messages, the reference message is merged with the i-th message, and the merged message is determined as the reference message, and the value of i is updated to i+1; When the i-th message does not exist in the sorted N messages, the loop is exited.

8. A message fusion device, characterized in that: include: A selection module, configured to select N messages from the M messages when M messages are obtained through N message channels and at least one message is obtained through each of the N message channels, wherein the message channels corresponding to the N messages are different from each other, N is an integer greater than or equal to 2, and M is an integer greater than or equal to N; The fusion module is used to fuse the N messages to obtain a fused message.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images