A robot's environmental perception method and decision-making system

Through multimodal data processing, robots can better perceive and understand their surroundings, generate individual descriptive text to guide action strategies, solve the problem of insufficient response of traditional robots in unknown environments, and achieve more comprehensive environmental perception and decision-making capabilities.

CN123103640APending Publication Date: 2026-11-17SHENZHEN EPS TECH CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611286328.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-24
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Traditional robot design relies on the configuration of pre-defined scenario conditions and parameters, which makes it unable to autonomously understand the environment and execute the correct response when the environment is not included.

Method used

By acquiring multimodal heterogeneous data of the robot's surrounding environment, extracting single-modal temporal feature data of each modality, performing temporal alignment of cross-modal data, generating a multimodal feature matrix, implementing spatial feature association, generating multimodal feature structure instances of environmental entities, and finally generating individual description text.

Benefits of technology

It enables robots to have a more comprehensive environmental perception capability for various types of unfamiliar environments, adapt to multiple environmental changes, and generate individual descriptive text to guide action strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123103640A_ABST
    Figure CN123103640A_ABST
Patent Text Reader

Abstract

This invention proposes a robot environmental perception method and decision-making system. It acquires multimodal heterogeneous data of the robot's surrounding environment, including visual, auditory, and tactile data collected by the robot from its surroundings through various sensing methods. Single-modal temporal feature data of each modality is extracted from the multimodal heterogeneous data. Cross-modal temporal alignment is performed on the single-modal temporal feature data to obtain a multimodal feature matrix. Spatial feature association is performed on the feature values ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment. Based on these multimodal feature structure instances, individual descriptive text for each environmental entity in the robot's surrounding environment is generated. This method possesses more comprehensive environmental perception capabilities and can adapt to various types of unfamiliar environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a robot's environmental perception method and decision-making system. Background Technology

[0002] The strength of a robot's environmental perception capability is a crucial indicator of its ability to interact with its environment. Specifically, the environmental data a robot can acquire through its sensors, such as cameras, radar, sound sensors, and tactile sensors, is the primary factor determining "what" a robot can do. Traditional robot design and control are reverse-engineered: we determine what the robot should be able to do, so we integrate sensors into the robot to collect corresponding environmental data. The robot then generates a corresponding control program based on the type, magnitude, strength, or other parameters of the collected environmental data, according to a pre-designed program. Therefore, the "intelligent" control of traditional robots essentially relies on the configuration of pre-defined scene parameters. The more conditions considered in the pre-defined scene and the more complex the parameter design, the more tasks the robot can "autonomously" handle. However, this design approach has significant drawbacks. When the robot encounters environmental conditions not included in its pre-defined scene, it cannot autonomously understand the environment and execute the correct response. Summary of the Invention

[0003] Based on the above problems, this invention proposes a robot environmental perception method and decision-making system, which has a more comprehensive environmental perception capability and can adapt to various types of unfamiliar environments.

[0004] In view of this, a first aspect of the present invention provides a method for robot environmental perception, comprising: Acquire multimodal heterogeneous data of the robot's surrounding environment, including visual data, auditory data, and tactile data collected by the robot from the surrounding environment through various sensing means; Extract single-mode temporal feature data of each modality from the multimodal heterogeneous data; Perform cross-modal data temporal alignment on the single-modal temporal feature data to obtain a multimodal feature matrix; Spatial domain feature association is performed on the eigenvalues ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment; Generate individual description texts for each environmental entity in the robot's surrounding environment based on multimodal feature structure instances.

[0005] Optionally, the step of extracting single-mode time-series feature data of each modality feature from the multimodal heterogeneous data specifically includes: Use a pre-configured feature extraction function to extract feature values ​​of each modality from multi-source heterogeneous data from different data sources; Determine whether the modal feature is an instantaneous feature or a stage feature; When the modal feature is an instantaneous feature, the acquisition time corresponding to the feature value of the modal feature in the multi-source heterogeneous data is determined as the association time of the feature value; When the modal feature is a stage feature, the start time, end time, or any time point between the two of the stage feature in the multi-source heterogeneous data is determined as the associated time of the corresponding feature value. Generate a key-value pair sequence for each modality feature, with the associated time and the feature value as key-value pairs; The key-value pair sequence is determined as the single-mode temporal feature data of the corresponding modality feature.

[0006] Optionally, the step of performing cross-modal data temporal alignment on the single-modal temporal feature data to obtain a multimodal feature matrix specifically includes: The single-mode temporal feature data is analyzed to determine the period of significant feature, which is the time period in the multi-source heterogeneous data collected by the robot where there are significant changes in modal features; Perform temporal alignment on the single-mode temporal series feature data within the significant period of the features; The multimodal feature matrix corresponding to the salient time period is generated using the time-domain aligned single-mode temporal feature data.

[0007] Optionally, the step of analyzing the single-mode time series feature data to determine the significant period of features specifically includes: Configuration feature comparison duration; Significant features are identified from the modal features extracted from the multimodal heterogeneous data, and the moment when the feature value of the significant feature crosses the boundary is determined as the target moment. Determine the associated features of the salient features in the multimodal vector space; Determine a target time range, wherein the target time range is a time range with the target time as the midpoint and a length equal to the feature comparison duration; The significant time period of the feature is determined based on the numerical changes of the associated feature within the target time range.

[0008] Optionally, the step of determining the associated features of the salient features in the multimodal vector space specifically includes: A multimodal vector space for analyzing feature similarity is constructed, wherein the number of feature dimensions in the multimodal vector space corresponds to the feature comparison duration; Local feature data that fall within the target time range are obtained from the single-mode temporal feature data of each modality feature to generate a local feature value sequence for each modality feature; The local feature data that fall within the target time range in the single-mode temporal feature data of each modality feature are mapped to the multimodal vector space to obtain the corresponding local vector; Calculate the cosine distance between the local vector of the salient feature and the local vectors of other modal features in the multimodal vector space; Modal features corresponding to local vectors whose cosine distance to the local vectors of the salient features is less than a preset distance threshold are identified as associated features.

[0009] Optionally, the step of determining the significant time period of the feature based on the numerical change of the correlation feature within the target time range specifically includes: Obtain the pre-configured significance coefficient And the salient boundaries corresponding to each association feature. The significance coefficients mentioned above , , It is a positive integer. The number of associated features of the salient feature; Each feature value in the local feature value sequence of the associated features is represented as: ,in , It is a positive integer. The first of the salient features The number of feature values ​​in the local feature value sequence of each associated feature; Generate a subsequence of local feature value sequences for each associated feature, and represent each feature value in the subsequence as follows: ,in , It is a positive integer. The first of the salient features The number of feature values ​​in a subsequence of a sequence of local feature values ​​of a related feature, and: ; Generate an associated time series corresponding to each feature value in the subsequence, and represent each time value in the associated time series as... ; Determine the first time in the associated time series And the second time So that the first time and the second time satisfy: ; Construct the sub-significant time range for each associated feature ; The sub-significant time range of each associated feature of the salient feature. The union of these values ​​is determined as the significant time period of the feature.

[0010] Optionally, the step of performing temporal domain alignment on the single-mode temporal series feature data within the period of significant feature occurrence specifically includes: The local feature data falling within the significant time period of the feature in the single-mode temporal feature data of the significant feature are constructed into a first feature sequence; Each of the associated features of the salient feature is used to construct a second feature sequence from the local feature data that falls within the salient time period of the feature in the single-mode temporal feature data of each associated feature; Configure the target sequence length, which is the longest sequence among the first feature sequence and the second feature sequence. times, of which A pre-configured positive integer greater than 1; Expand the first feature sequence and the second feature sequence to the target sequence length; In the Transformer architecture, an attention mechanism is used to generate the alignment weight matrix between the first feature sequence and each second feature sequence; The first feature sequence and the second feature sequence are aligned based on the alignment weight matrix.

[0011] Optionally, the multimodal heterogeneous data includes environmental images of the robot's surrounding environment. The specific steps of performing spatial feature association on the feature values ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment include: Identify the salient features in the multimodal feature matrix corresponding to the feature events during the salient period of the feature; The environmental image of the robot's surrounding environment during the period of significant features is obtained from the multimodal heterogeneous data, and the environmental image is segmented into several environmental entities. Locate the spatial position of the feature event; The multimodal feature matrix is ​​associated with the corresponding environmental entities based on the spatial location of the feature events; Obtain the feature attribute list of the environmental entity, and match the feature values ​​in the multimodal feature matrix with the feature attributes in the feature attribute list; The multimodal feature structure of the environmental entity is constructed by using the name of the characteristic attribute of the environmental entity as the attribute name and the corresponding characteristic value as the attribute value.

[0012] Optionally, the step of obtaining environmental images of the robot's surrounding environment during the period of significant features from the multimodal heterogeneous data, and performing environmental entity segmentation on the environmental images to obtain several environmental entities, specifically includes: Perform comprehensive object recognition on the environmental image to obtain the name and location of each object in the image; Calculate the distance between the object and the robot based on the object's position; The relationship between pre-configured distance and independent recognition coefficient is obtained to determine the independent recognition coefficient of each object in the environmental image, wherein the independent recognition coefficient is inversely correlated with distance; The size of each object is calculated using the environmental image; The product of the size of an object and its corresponding independent recognition coefficient is used to determine the independent recognition score of the corresponding object. Objects with independent recognition scores greater than a preset threshold are identified as environmental entities.

[0013] A second aspect of the present invention provides an environmental perception system for a robot, comprising: The sensing module includes a data acquisition submodule that acquires environmental data through sensors, and a data processing submodule that processes the environmental data. The artificial intelligence module includes a text generation submodule for generating individual descriptive text, and a sequence expansion submodule for performing feature sequence expansion; The decision-making module includes an impact assessment submodule that determines the degree of influence of the corresponding environmental entity on the robot based on the individual description text, and a strategy generation submodule for generating robot action strategies. The decision system is configured to implement the environmental perception method according to any one of the first aspects of the present invention.

[0014] This invention proposes a robot environmental perception method and decision-making system. It acquires multimodal heterogeneous data of the robot's surrounding environment, including visual, auditory, and tactile data collected by the robot from its surroundings through various sensing methods. Single-modal temporal feature data of each modality is extracted from the multimodal heterogeneous data. Cross-modal temporal alignment is performed on the single-modal temporal feature data to obtain a multimodal feature matrix. Spatial feature association is performed on the feature values ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment. Based on these multimodal feature structure instances, individual descriptive text for each environmental entity in the robot's surrounding environment is generated. This method possesses more comprehensive environmental perception capabilities and can adapt to various types of unfamiliar environments. Attached Figure Description

[0015] Figure 1This is a flowchart of an environmental perception method for a robot provided in one embodiment of the present invention; Figure 2 This is a schematic diagram of an environmental perception system for a robot provided in one embodiment of the present invention. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0017] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0018] In the description of this invention, the term "multiple" refers to two or more. Unless otherwise explicitly defined, the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. The terms "connect," "install," "fix," etc., should be interpreted broadly. For example, "connect" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "multiple" means two or more.

[0019] In the description of this specification, the terms "one embodiment," "some implementations," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0020] The following description, with reference to the accompanying drawings, illustrates a robot's environmental perception method and decision-making system according to some embodiments of the present invention.

[0021] like Figure 1 As shown, the first aspect of the present invention proposes a method for robot environmental perception, comprising: Acquire multimodal heterogeneous data of the robot's surrounding environment, including visual data, auditory data, and tactile data collected by the robot from the surrounding environment through various sensing means; Extract single-mode temporal feature data of each modality from the multimodal heterogeneous data; Perform cross-modal data temporal alignment on the single-modal temporal feature data to obtain a multimodal feature matrix; Spatial domain feature association is performed on the eigenvalues ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment; Generate individual description texts for each environmental entity in the robot's surrounding environment based on multimodal feature structure instances.

[0022] Specifically, the multimodal heterogeneous data refers to various types of sensory data with different data structures and representations collected by the robot from the environment through multiple sensing methods. In terms of data representation, it can be data types of different modalities such as text, images, audio, and video. At the data level where the robot simulates human senses, it can also be divided into different modalities such as visual data, auditory data, tactile data, and other types of sensory data.

[0023] The single-mode temporal feature data is a time data sequence obtained by converting single-mode sensor data into one or more feature values, and then sampling and arranging these feature values ​​in chronological order. Therefore, depending on the feature type of its corresponding modal feature, the single-mode temporal feature data can be a single time data sequence or several time data sequences. That is, a modal data may contain several types of modal features, and the feature data of each modal feature can be constructed into a time data sequence. For example, for tactile data, it can include multiple feature types such as pressure features, vibration intensity features, and vibration frequency features. Similarly, for auditory data, it can include multiple feature types such as sound intensity features and sound frequency features.

[0024] A time data sequence consists of several feature elements arranged in chronological order. In some implementations, a feature element is a "time, feature value" key-value pair with time as the key and feature value as the value.

[0025] The multimodal feature matrix is ​​a feature matrix obtained by aligning and arranging the feature values ​​of several modal features of each modality along the time dimension.

[0026] The environmental entity is an independent individual identified by specific object recognition rules from people or objects in the robot's surrounding environment.

[0027] Furthermore, the steps for generating individual descriptive text for each environmental entity in the robot's surrounding environment based on multimodal feature structure instances specifically include: The multimodal feature structure instance is constructed as an input vector; The input vector is fed into a pre-trained text generation model to generate the individual description text.

[0028] The multimodal feature structure instance is composed of several feature attributes and their corresponding feature values. In the step of constructing the multimodal feature structure instance into an input vector, the feature values ​​of each feature attribute are sorted in a preset order to obtain a feature value sequence, and this feature value sequence is mapped to a vector space to obtain the input vector. The text generation model is a deep learning model trained with a large amount of sample data.

[0029] Furthermore, after the step of generating individual description texts for each environmental entity in the robot's surrounding environment based on multimodal feature structure instances, the process also includes: The degree of influence of the corresponding environmental entity on the robot is determined based on the individual description text. When the influence of the environmental entity on the robot is greater than a preset influence threshold, a robot action strategy corresponding to the individual description text is generated.

[0030] Using the technical solution of the above implementation method, when the influence of multiple environmental entities around the robot on the robot is greater than the influence threshold, the robot will generate several identical or different action strategies. When there are several different action strategies, the robot needs to judge the conflict and importance of these action strategies, and then select one or more action strategies to generate its motion parameters.

[0031] Furthermore, in the step of using the individual description text, an artificial intelligence model can be trained to determine the degree of influence of environmental entities on the robot. The artificial intelligence model takes the vectorized individual description text as input data and the influence degree value within a specific range (e.g., 0-1, or 0-100) as output data.

[0032] Optionally, the step of extracting single-mode time-series feature data of each modality feature from the multimodal heterogeneous data specifically includes: Use a pre-configured feature extraction function to extract feature values ​​of each modality from multi-source heterogeneous data from different data sources; Determine whether the modal feature is an instantaneous feature or a stage feature; When the modal feature is an instantaneous feature, the acquisition time corresponding to the feature value of the modal feature in the multi-source heterogeneous data is determined as the association time of the feature value; When the modal feature is a stage feature, the start time, end time, or any time point between the two of the stage feature in the multi-source heterogeneous data is determined as the associated time of the corresponding feature value. Generate a key-value pair sequence for each modality feature, with the associated time and the feature value as key-value pairs; The key-value pair sequence is determined as the single-mode temporal feature data of the corresponding modality feature.

[0033] Specifically, the data source refers to the sensors that collect the various groups of data from the multi-source heterogeneous data. Different sensors are used to collect data, resulting in different data types and structures. For data from different sensors, corresponding feature extraction functions are configured to extract feature values ​​of one or more modal features.

[0034] In the technical solutions of the above embodiments, the instantaneous feature is a feature reflecting the instantaneous state of a certain thing in the environment at a certain moment, such as pressure features, vibration intensity features, sound intensity features, etc., which are all instantaneous features. The stage feature is a feature reflecting the stage state of a certain thing in the environment during a certain time period, such as average features such as average velocity features, average vibration intensity features, etc., and periodic features such as oscillation period, rotation period, etc.

[0035] Optionally, the step of performing cross-modal data temporal alignment on the single-modal temporal feature data to obtain a multimodal feature matrix specifically includes: The single-mode temporal feature data is analyzed to determine the period of significant feature, which is the time period in the multi-source heterogeneous data collected by the robot where there are significant changes in modal features; Perform temporal alignment on the single-mode temporal series feature data within the significant period of the features; The multimodal feature matrix corresponding to the salient time period is generated using the time-domain aligned single-mode temporal feature data.

[0036] It should be understood that the generation time of data from different modalities is often inconsistent. Furthermore, since different modal data are collected independently by different sensors, their data acquisition cycles will also differ. Therefore, the generation time of the multimodal environmental data perceived by the robot will deviate somewhat from the actual occurrence time of the environmental events. The data generation time refers to the time it takes for the state or change corresponding to a specific event in the environment to reach the robot's corresponding sensors through sound, light, vibration, or other means. For example, there may be a delay between a robot capturing an image of an object moving via a camera and capturing the sound emitted by that object due to movement via a microphone.

[0037] In the technical solution of the above embodiments, the significant feature period is a time period in which one or more feature values ​​extracted from the multi-source heterogeneous data collected by the robot change significantly when one or more objects in the environment move, emit sounds, or change their appearance. Temporal alignment is performed on the single-mode temporal sequence feature data only when there are significantly changed features in the multi-source heterogeneous data collected by the robot.

[0038] Optionally, the step of analyzing the single-mode time series feature data to determine the significant period of features specifically includes: Configuration feature comparison duration; Significant features are identified from the modal features extracted from the multimodal heterogeneous data, and the moment when the feature value of the significant feature crosses the boundary is determined as the target moment. Determine the associated features of the salient features in the multimodal vector space; Determine a target time range, wherein the target time range is a time range with the target time as the midpoint and a length equal to the feature comparison duration; The significant time period of the feature is determined based on the numerical changes of the associated feature within the target time range.

[0039] Furthermore, the step of determining salient features from the modal features extracted from the multimodal heterogeneous data specifically includes: Obtain the salient boundaries corresponding to each pre-configured modal feature; Iterate through the single-mode temporal feature data for each modality feature; Determine whether there are out-of-bounds values ​​in the single-mode time series feature data. The out-of-bounds value is a feature value in the single-mode time series feature that is greater than the upper boundary value of the significant boundary, or a feature value that is less than the lower boundary value of the significant boundary. When an out-of-bounds value exists in the single-mode time-series feature data of any modal feature, the modal feature is determined as the significant feature.

[0040] In some implementations, depending on the feature type, the salient boundary may include an upper boundary value and / or a lower boundary value. The step of determining whether there are out-of-bounds values ​​in the single-mode time-series feature data includes determining whether there are feature values ​​in the single-mode time-series feature data that are greater than the upper boundary value or less than the lower boundary value. For some modal features, their salient boundary only includes the upper boundary value; a feature is only determined to be salient if its feature value is greater than the upper boundary value. For other modal features, their boundary values ​​include both an upper boundary value and a lower boundary value; a feature value greater than its upper boundary value and less than its lower boundary value are both determined to be salient features.

[0041] Optionally, the step of determining the associated features of the salient features in the multimodal vector space specifically includes: A multimodal vector space for analyzing feature similarity is constructed, wherein the number of feature dimensions in the multimodal vector space corresponds to the feature comparison duration; Local feature data that fall within the target time range are obtained from the single-mode temporal feature data of each modality feature to generate a local feature value sequence for each modality feature; The local feature data that fall within the target time range in the single-mode temporal feature data of each modality feature are mapped to the multimodal vector space to obtain the corresponding local vector; Calculate the cosine distance between the local vector of the salient feature and the local vectors of other modal features in the multimodal vector space; Modal features corresponding to local vectors whose cosine distance to the local vectors of the salient features is less than a preset distance threshold are identified as associated features.

[0042] In the technical solution of the above embodiments, the multimodal vector space is a vector space jointly composed of the modal features of each modality. The local vector is a normalized feature vector. That is, in the step of mapping the local feature data falling within the target time range in the single-modal time series feature data of each modality feature to the multimodal vector space to obtain the corresponding local vector, it is necessary to process the local feature data into normalized values ​​before mapping it to the multimodal vector space to generate the local vector. This avoids the distance calculation results being affected by the differences in the dimensions and numerical values ​​of different modal feature data.

[0043] Furthermore, the step of mapping the local feature data falling within the target time range from the single-modal temporal feature data of each modality to the multimodal vector space to obtain the corresponding local vector also includes refitting and sampling the single-modal temporal feature data so that the feature value sequences used to generate the local vectors for each modality have the same length. Since different modality features have different types, their feature value sampling or calculation intervals often vary significantly. To map them to a unified vector space, it is necessary to obtain feature value sequences of the same length for each modality within the target time range through fitting resampling or other processing methods.

[0044] Optionally, the step of determining the significant time period of the feature based on the numerical change of the correlation feature within the target time range specifically includes: Obtain the pre-configured significance coefficient And the salient boundaries corresponding to each association feature. The significance coefficients mentioned above , , It is a positive integer. The number of associated features of the salient feature; Each feature value in the local feature value sequence of the associated features is represented as: ,in , It is a positive integer. The first of the salient features The number of feature values ​​in the local feature value sequence of each associated feature; Generate a subsequence of local feature value sequences for each associated feature, and represent each feature value in the subsequence as follows: ,in , It is a positive integer. The first of the salient features The number of feature values ​​in a subsequence of a sequence of local feature values ​​of a related feature, and: ; Generate an associated time series corresponding to each feature value in the subsequence, and represent each time value in the associated time series as... ; Determine the first time in the associated time series And the second time So that the first time and the second time satisfy: ; Construct the sub-significant time range for each associated feature ; The sub-significant time range of each associated feature of the salient feature. The union of these values ​​is determined as the significant time period of the feature.

[0045] In the technical solution of the above embodiments, the number of feature values ​​in the subsequence of the local feature value sequence of each associated feature of the salient feature is... For the local feature value sequence that satisfies The number of feature values. Each time in the associated time series is the associated time corresponding to each feature value in the subsequence. Therefore, the length of the associated time series is the same as the length of the subsequence, and the associated time in the associated time series corresponds one-to-one with the feature value in the subsequence.

[0046] In practice, a uniform significance coefficient can be configured for all modal features, or independent significance coefficients can be configured for each modal feature. Since the amplitude range of background noise corresponding to the feature values ​​of different modal features often varies, the approach of configuring independent significance coefficients for each modal feature is superior for scenarios requiring high precision in configuring the feature saliency period. However, in most scenarios, especially when system resource constraints are limited, lower precision in the feature saliency period is more beneficial for reducing system deployment difficulty and simplifying the system deployment process.

[0047] Optionally, the step of performing temporal domain alignment on the single-mode temporal series feature data within the period of significant feature occurrence specifically includes: The local feature data falling within the significant time period of the feature in the single-mode temporal feature data of the significant feature are constructed into a first feature sequence; Each of the associated features of the salient feature is used to construct a second feature sequence from the local feature data that falls within the salient time period of the feature in the single-mode temporal feature data of each associated feature; Configure the target sequence length, which is the longest sequence among the first feature sequence and the second feature sequence. times, of which A pre-configured positive integer greater than 1; Expand the first feature sequence and the second feature sequence to the target sequence length; In the Transformer architecture, an attention mechanism is used to generate the alignment weight matrix between the first feature sequence and each second feature sequence; The first feature sequence and the second feature sequence are aligned based on the alignment weight matrix.

[0048] In the step of constructing a second feature sequence from the local feature data falling within the salient time period of each associated feature of the salient feature in the single-mode time series feature data of each associated feature, each associated feature corresponds to a second feature sequence, that is, the number of second feature sequences is the same as the number of associated features.

[0049] In the above embodiments, The expansion factor is a pre-configured factor; the larger the value of the expansion factor, the higher the accuracy of temporal alignment of the single-mode time-series feature data within the significant feature period. In the step of expanding the first feature sequence and the second feature sequence to the target sequence length, a relatively simple implementation is to expand the feature sequences by function fitting and resampling or by using an interpolation algorithm. Of course, sequence expansion can also be implemented by training a dedicated sequence expansion neural network model using deep learning. To ensure the numerical accuracy of the sequence expansion, using a dedicated sequence expansion neural network model is a better approach.

[0050] For ease of implementation, the above embodiment expands each associated feature into an expanded sequence of the same length as the expanded sequence of the salient feature. However, in actual implementation, the expanded sequences of each associated feature and the salient feature can be of different lengths. Furthermore, if an associated feature or the salient feature itself already has a high sampling rate, it may not need to be expanded.

[0051] In the step of generating the alignment weight matrix of the first feature sequence and each of the second feature sequences, the alignment weight matrix reflects the attention (i.e., weight value) of each feature element of the second feature sequence to each feature element of the first feature sequence, and the point with the highest attention is the alignment point of the feature elements in the two sequences.

[0052] Furthermore, generating the multimodal feature matrix corresponding to the salient time period using the time-domain aligned single-mode temporal series feature data specifically includes: Determine the offset between the first feature sequence and the second feature sequence during the alignment process; According to the offset, fill the beginning / end of the first feature sequence or the second feature sequence with alignment feature values ​​so that the first feature sequence and the second feature sequence still have the same number of feature values ​​during the feature significant period after alignment; The first feature sequence and the second feature sequences of each associated feature are merged into the multimodal feature matrix.

[0053] Optionally, the multimodal heterogeneous data includes environmental images of the robot's surrounding environment. The specific steps of performing spatial feature association on the feature values ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment include: Identify the salient features in the multimodal feature matrix corresponding to the feature events during the salient period of the feature; The environmental image of the robot's surrounding environment during the period of significant features is obtained from the multimodal heterogeneous data, and the environmental image is segmented into several environmental entities. Locate the spatial position of the feature event; The multimodal feature matrix is ​​associated with the corresponding environmental entities based on the spatial location of the feature events; Obtain the feature attribute list of the environmental entity, and match the feature values ​​in the multimodal feature matrix with the feature attributes in the feature attribute list; The multimodal feature structure of the environmental entity is constructed by using the name of the characteristic attribute of the environmental entity as the attribute name and the corresponding characteristic value as the attribute value.

[0054] Specifically, when different modal features are used as salient features, they are often associated with one or more specific feature events. In some implementation solutions, the types of salient features and their associated features, as well as the feature events associated with their feature value ranges, can be configured. Thus, after performing cross-modal data temporal alignment on the single-modal time-series feature data to obtain a multimodal feature matrix, the feature events corresponding to the salient features in the multimodal feature matrix within the salient period can be quickly determined based on the association relationship.

[0055] In the step of locating the spatial position of the feature event, its spatial position can be determined using the salient features associated with the feature event and the feature values ​​of its associated features. For example, when one of the salient features associated with the feature event and its associated features is an image feature, the spatial position of the feature event can be determined by the position of the corresponding image feature in the environmental image. Similarly, when one of the salient features associated with the feature event and its associated features is a sound feature, the spatial position of the feature event can be determined based on the sound intensity, the relative azimuth angle of the sound source, and other feature values ​​of the sound feature. After determining the spatial position of the feature event, it can be associated with environmental entities within a certain spatial range.

[0056] Optionally, the step of obtaining environmental images of the robot's surrounding environment during the period of significant features from the multimodal heterogeneous data, and performing environmental entity segmentation on the environmental images to obtain several environmental entities, specifically includes: Perform comprehensive object recognition on the environmental image to obtain the name and location of each object in the image; Calculate the distance between the object and the robot based on the object's position; The relationship between pre-configured distance and independent recognition coefficient is obtained to determine the independent recognition coefficient of each object in the environmental image, wherein the independent recognition coefficient is inversely correlated with distance; The size of each object is calculated using the environmental image; The product of the size of an object and its corresponding independent recognition coefficient is used to determine the independent recognition score of the corresponding object. Objects with independent recognition scores greater than a preset threshold are identified as environmental entities.

[0057] Specifically, in the step of performing comprehensive object recognition on the environmental image to obtain the names and locations of each object in the image, all identifiable objects in the image are identified and their locations are calculated. All identifiable objects include objects with inclusion relationships, such as buildings and their windows, plants and their leaves or flowers, cars and their tires, or human bodies and their ornaments. The larger the object and the more image details it has in the environmental image, the deeper its possible inclusion hierarchy.

[0058] In the step of obtaining the pre-configured relationship between distance and independent recognition coefficient to determine the independent recognition coefficient of each object in the environmental image, the inverse correlation between the independent recognition coefficient and distance means that the greater the distance between the object and the robot, the smaller its independent recognition coefficient.

[0059] Depending on the implementation needs, such as the accuracy or difficulty of the calculation, in the step of calculating the size of each object through the environmental image, any one of the object's volume, the area of ​​the visible region, its length, or its width can be selected as the object's size.

[0060] like Figure 2 As shown, a second aspect of the present invention provides an environmental perception system for a robot, comprising: The sensing module includes a data acquisition submodule that acquires environmental data through sensors, and a data processing submodule that processes the environmental data. The artificial intelligence module includes a text generation submodule for generating individual descriptive text, and a sequence expansion submodule for performing feature sequence expansion; The decision-making module includes an impact assessment submodule that determines the degree of influence of the corresponding environmental entity on the robot based on the individual description text, and a strategy generation submodule for generating robot action strategies. The decision system is configured to implement the environmental perception method according to any one of the first aspects of the present invention.

[0061] Specifically, the data acquisition submodule acquires environmental image data, sound data, and other types of sensor data through sensors such as cameras, microphones, and others. The data processing submodule performs processing on the environmental data, such as noise reduction to remove abnormal data, and standardization processing such as unit unification, and constructs the acquired data into the multi-source heterogeneous data according to the encoding rules of the corresponding sensor data.

[0062] The AI ​​module is responsible for the interaction between the multi-source heterogeneous data collected by the robot, intermediate data during processing, and the AI ​​model. In practice, the trained AI model can be deployed in the cloud, and the AI ​​module connects to the cloud server via a network to interface with the corresponding AI model's data interface. For example, the module submits the feature sequences that need to be expanded to the corresponding AI model deployed on the cloud server via the network and receives the expanded feature sequences returned by the model, which are then used for subsequent feature sequence alignment.

[0063] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0064] As described above, these embodiments of the present invention do not exhaustively cover all details, nor do they limit the invention to the specific embodiments described. Clearly, many modifications and variations can be made based on the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to effectively utilize the invention and its modifications. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for environmental perception of a robot, characterized in that, include: Acquire multimodal heterogeneous data of the robot's surrounding environment, including visual data, auditory data, and tactile data collected by the robot from the surrounding environment through various sensing means; Extract single-mode temporal feature data of each modality from the multimodal heterogeneous data; Perform cross-modal data temporal alignment on the single-modal temporal feature data to obtain a multimodal feature matrix; Spatial domain feature association is performed on the eigenvalues ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment; Generate individual description texts for each environmental entity in the robot's surrounding environment based on multimodal feature structure instances.

2. The robot environmental perception method according to claim 1, characterized in that, The steps for extracting single-mode time-series feature data of each modality from the multimodal heterogeneous data specifically include: Use a pre-configured feature extraction function to extract feature values ​​of each modality from multi-source heterogeneous data from different data sources; Determine whether the modal feature is an instantaneous feature or a stage feature; When the modal feature is an instantaneous feature, the acquisition time corresponding to the feature value of the modal feature in the multi-source heterogeneous data is determined as the association time of the feature value; When the modal feature is a stage feature, the start time, end time, or any time point between the two of the stage feature in the multi-source heterogeneous data is determined as the associated time of the corresponding feature value. Generate a key-value pair sequence for each modality feature, with the associated time and the feature value as key-value pairs; The key-value pair sequence is determined as the single-mode temporal feature data of the corresponding modality feature.

3. The robot environmental perception method according to claim 1, characterized in that, The steps of performing cross-modal data temporal alignment on the single-modal temporal feature data to obtain a multimodal feature matrix specifically include: The single-mode temporal feature data is analyzed to determine the period of significant feature, which is the time period in the multi-source heterogeneous data collected by the robot where there are significant changes in modal features; Perform temporal alignment on the single-mode temporal series feature data within the significant period of the features; The multimodal feature matrix corresponding to the salient time period is generated using the time-domain aligned single-mode temporal feature data.

4. The robot environmental perception method according to claim 3, characterized in that, The steps for analyzing the single-mode time series feature data to determine the significant feature periods specifically include: Configuration feature comparison duration; Significant features are identified from the modal features extracted from the multimodal heterogeneous data, and the moment when the feature value of the significant feature crosses the boundary is determined as the target moment. Determine the associated features of the salient features in the multimodal vector space; Determine a target time range, wherein the target time range is a time range with the target time as the midpoint and a length equal to the feature comparison duration; The significant time period of the feature is determined based on the numerical changes of the associated feature within the target time range.

5. The robot environmental perception method according to claim 4, characterized in that, The steps for determining the associated features of the salient features in the multimodal vector space specifically include: A multimodal vector space for analyzing feature similarity is constructed, wherein the number of feature dimensions in the multimodal vector space corresponds to the feature comparison duration; Local feature data that fall within the target time range are obtained from the single-mode temporal feature data of each modality feature to generate a local feature value sequence for each modality feature; The local feature data that fall within the target time range in the single-mode temporal feature data of each modality feature are mapped to the multimodal vector space to obtain the corresponding local vector; Calculate the cosine distance between the local vector of the salient feature and the local vectors of other modal features in the multimodal vector space; Modal features corresponding to local vectors whose cosine distance to the local vectors of the salient features is less than a preset distance threshold are identified as associated features.

6. The robot environmental perception method according to claim 5, characterized in that, The step of determining the significant time period of the feature based on the numerical change of the correlation feature within the target time range specifically includes: Obtain the pre-configured significance coefficient And the salient boundaries corresponding to each association feature. The significance coefficients mentioned above , , It is a positive integer. The number of associated features of the salient feature; Each feature value in the local feature value sequence of the associated features is represented as: ,in , It is a positive integer. The first of the salient features The number of feature values ​​in the local feature value sequence of each associated feature; Generate a subsequence of local feature value sequences for each associated feature, and represent each feature value in the subsequence as follows: ,in , It is a positive integer. The first of the salient features The number of feature values ​​in a subsequence of a sequence of local feature values ​​of a related feature, and: ; Generate an associated time series corresponding to each feature value in the subsequence, and represent each time value in the associated time series as... ; Determine the first time in the associated time series And the second time So that the first time and the second time satisfy: ; Construct the sub-significant time range for each associated feature ; The sub-significant time range of each associated feature of the salient feature. The union of these values ​​is determined as the significant time period of the feature.

7. The robot environmental perception method according to claim 3, characterized in that, The steps for temporal domain alignment of single-mode temporal series feature data within the salient period of the features specifically include: The local feature data falling within the significant time period of the feature in the single-mode temporal feature data of the significant feature are constructed into a first feature sequence; Each of the associated features of the salient feature is used to construct a second feature sequence from the local feature data that falls within the salient time period of the feature in the single-mode temporal feature data of each associated feature; Configure the target sequence length, which is the longest sequence among the first feature sequence and the second feature sequence. times, of which A pre-configured positive integer greater than 1; Expand the first feature sequence and the second feature sequence to the target sequence length; In the Transformer architecture, an attention mechanism is used to generate the alignment weight matrix between the first feature sequence and each second feature sequence; The first feature sequence and the second feature sequence are aligned based on the alignment weight matrix.

8. The robot environmental perception method according to claim 1, characterized in that, The multimodal heterogeneous data includes environmental images of the robot's surrounding environment. The specific steps of performing spatial domain feature association on the feature values ​​in the multimodal feature matrix to generate multimodal feature structure instances of various environmental entities in the robot's surrounding environment include: Identify the salient features in the multimodal feature matrix corresponding to the feature events during the salient period of the feature; The environmental image of the robot's surrounding environment during the period of significant features is obtained from the multimodal heterogeneous data, and the environmental image is segmented into several environmental entities. Locate the spatial position of the feature event; The multimodal feature matrix is ​​associated with the corresponding environmental entities based on the spatial location of the feature events; Obtain the feature attribute list of the environmental entity, and match the feature values ​​in the multimodal feature matrix with the feature attributes in the feature attribute list; The multimodal feature structure of the environmental entity is constructed by using the name of the characteristic attribute of the environmental entity as the attribute name and the corresponding characteristic value as the attribute value.

9. The robot environmental perception method according to claim 8, characterized in that, The steps of obtaining environmental images of the robot's surrounding environment during the period of significant features from the multimodal heterogeneous data, and performing environmental entity segmentation on the environmental images to obtain several environmental entities, specifically include: Perform comprehensive object recognition on the environmental image to obtain the name and location of each object in the image; Calculate the distance between the object and the robot based on the object's position; The relationship between pre-configured distance and independent recognition coefficient is obtained to determine the independent recognition coefficient of each object in the environmental image, wherein the independent recognition coefficient is inversely correlated with distance; The size of each object is calculated using the environmental image; The product of the size of an object and its corresponding independent recognition coefficient is used to determine the independent recognition score of the corresponding object. Objects with independent recognition scores greater than a preset threshold are identified as environmental entities.

10. A decision-making system for a robot, characterized in that, include: The sensing module includes a data acquisition submodule that acquires environmental data through sensors, and a data processing submodule that processes the environmental data. The artificial intelligence module includes a text generation submodule for generating individual descriptive text, and a sequence expansion submodule for performing feature sequence expansion; The decision-making module includes an impact assessment submodule that determines the degree of influence of the corresponding environmental entity on the robot based on the individual description text, and a strategy generation submodule for generating robot action strategies. The decision system is configured to implement the environmental perception method as described in any one of claims 1-9.