Visual perception processing method and device, electronic equipment and storage medium

Abstract

The application provides a visual perception processing method and device, electronic equipment and storage medium, the method comprises: acquiring the first viewpoint image and the second viewpoint image of the target object at the current time; by the improved spatial feature extraction model, the spatial feature of the first viewpoint image is extracted to obtain the first spatial feature, and the spatial feature of the second viewpoint image is extracted to obtain the second spatial feature; by the time sequence feature extraction model, the first spatial feature, the second spatial feature and the hidden feature of the last time of the current time are used to obtain the time sequence feature and the hidden feature of the current time, so as to make decisions based on the time sequence feature. The physical coverage range of the environment perception of the viewpoint image obtained by the wide-angle visual sensor is larger and more comprehensive, the time sequence feature is extracted after the spatial feature extraction, the continuity of the time dimension is considered, the instantaneous perception robustness is improved, and the obtained time sequence feature is more accurate.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a visual perception processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In intelligent systems such as service robots, industrial collaborative robots, and autonomous operating platforms, visual perception, as a core environmental interaction modality, has become a key input source for high-level decision-making models such as navigation planning, object manipulation, and human-machine collaboration due to its significant advantages such as low cost, rich information dimensions, no need to actively emit signals, and strong task generalization ability.

[0003] Existing visual sensors generally use single-viewpoint ordinary sensors, which limits the field of view perceived by such visual perception methods. Furthermore, existing visual perception models usually only focus on spatial feature extraction, resulting in poor visual perception results. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a visual perception processing method, apparatus, electronic device, and storage medium to improve the accuracy of visual perception processing.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a visual perception processing method, the method comprising: Acquire a first viewpoint image and a second viewpoint image of the target object at the current moment. The first viewpoint image is acquired by a first wide-angle vision sensor, and the second viewpoint image is acquired by a second wide-angle vision sensor. The first viewpoint image and the second viewpoint image are respectively input into the improved spatial feature extraction model. The improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain the first spatial feature, and extracts spatial features from the second viewpoint image to obtain the second spatial feature. The first spatial feature and the second spatial feature have the same spatial dimension. Both the first spatial feature and the second spatial feature are input into the temporal feature extraction model. The temporal feature extraction model obtains the temporal features and hidden features of the current moment based on the first spatial feature, the second spatial feature, and the hidden features of the previous moment of the current moment, so as to make decisions based on the temporal features.

[0006] Optionally, the improved spatial feature extraction model includes an initial convolutional pooling layer, a standard intermediate layer, a channel-locking layer, a pooling mapping layer, and an identity mapping layer connected in sequence, wherein the channel-locking layer keeps the number of channels unchanged when the feature map is downsampled.

[0007] Optionally, the step of extracting spatial features from the first viewpoint image using the improved spatial feature extraction model to obtain the first spatial features includes: The first viewpoint image is input into the improved spatial feature extraction model, and spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locked layer, pooling mapping layer and identity mapping layer in the spatial feature extraction model to obtain the first spatial feature.

[0008] Optionally, the spatial feature extraction, which sequentially passes through the initial convolutional pooling layer, standard intermediate layer, channel-locking layer, pooling mapping layer, and identity mapping layer in the spatial feature extraction model to obtain the first spatial feature, includes: The first viewpoint image is input into the initial convolutional pooling layer, and the initial convolutional pooling layer performs convolution and downsampling processing on the first viewpoint image to obtain the first intermediate feature map; The first intermediate feature map is input into the standard intermediate layer, and the standard intermediate layer performs downsampling processing on the first intermediate feature map to obtain the second intermediate feature map. The second intermediate feature map is input to the channel locking layer, and the channel locking layer performs downsampling processing on the second intermediate feature map based on a fixed number of channels to obtain the third intermediate feature map; The third intermediate feature map is input into the pooling mapping layer, and the pooling mapping layer compresses the third intermediate feature map to obtain the fourth intermediate feature map. The fourth intermediate feature map is input to the identity mapping layer, which performs an identity mapping on the fourth intermediate feature map and outputs the fourth intermediate feature map as the first spatial feature.

[0009] Optionally, the step of obtaining the temporal features and hidden features of the current moment by the temporal feature extraction model based on the first spatial features, the second spatial features, and the hidden features of the previous moment of the current moment includes: The first spatial feature, the second spatial feature, and the hidden feature from the previous time step are input into the temporal feature extraction model, and then processed sequentially by the feature splicing module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal feature and hidden feature at the current time step.

[0010] Optionally, the processing, sequentially via the feature concatenation module, the loop processing module, and the compression module in the temporal feature extraction model, to obtain the temporal features and hidden features at the current time, includes: The first spatial feature and the second spatial feature are input into the feature concatenation module in the temporal feature extraction model. The feature concatenation module concatenates the first spatial feature and the second spatial feature to obtain the concatenated feature. The splicing features and the hidden features of the previous time step are input into the loop module of the temporal feature extraction model. The loop module performs temporal feature extraction based on the splicing features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step. The initial time-series features at the current moment are input into the feature compression module in the time-series feature extraction model. The feature compression module compresses the initial time-series features to obtain the time-series features at the current moment.

[0011] Optionally, the loop module includes a first gated loop and a second gated loop layer connected in sequence; The step of extracting temporal features by the loop module based on the splicing features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step includes: The splicing features and the first hidden features of the previous time step are input into the first gated loop layer. The first gated loop layer performs temporal feature extraction on the splicing features and the hidden features to obtain the first temporal features of the current time step and the first hidden features of the current time step. The first temporal feature and the first hidden feature at the current moment are input into the second gated loop layer. The second gated loop layer extracts temporal features from the first temporal feature and the second hidden feature at the previous moment to obtain the initial temporal feature and the second hidden feature at the current moment. The first hidden feature and the second hidden feature at the current moment are used as the hidden feature at the current moment.

[0012] Secondly, embodiments of this application also provide a visual perception processing device, including: The acquisition module is used to acquire a first viewpoint image and a second viewpoint image of the target object at the current moment. The first viewpoint image is acquired by a first wide-angle vision sensor, and the second viewpoint image is acquired by a second wide-angle vision sensor. The first feature extraction module is used to input the first viewpoint image and the second viewpoint image into the improved spatial feature extraction model, and the improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain a first spatial feature, and extracts spatial features from the second viewpoint image to obtain a second spatial feature, wherein the first spatial feature and the second spatial feature have the same dimension. The second feature extraction module is used to input both the first spatial feature and the second spatial feature into the temporal feature extraction model. The temporal feature extraction module obtains the temporal feature and hidden feature of the current moment based on the first spatial feature, the second spatial feature, and the hidden feature of the previous moment of the current moment, so as to make a decision based on the temporal feature and hidden feature.

[0013] Optionally, the improved spatial feature extraction model includes an initial convolutional pooling layer, a standard intermediate layer, a channel-locking layer, a pooling mapping layer, and an identity mapping layer connected in sequence, wherein the channel-locking layer keeps the number of channels unchanged when the feature map is downsampled.

[0014] Optionally, the first feature extraction module is specifically used for: The first viewpoint image is input into the improved spatial feature extraction model, and spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locked layer, pooling mapping layer and identity mapping layer in the spatial feature extraction model to obtain the first spatial feature.

[0015] Optionally, the first feature extraction module is specifically used for: The first viewpoint image is input into the initial convolutional pooling layer, and the initial convolutional pooling layer performs convolution and downsampling processing on the first viewpoint image to obtain the first intermediate feature map; The first intermediate feature map is input into the standard intermediate layer, and the standard intermediate layer performs downsampling processing on the first intermediate feature map to obtain the second intermediate feature map. The second intermediate feature map is input to the channel locking layer, and the channel locking layer performs downsampling processing on the second intermediate feature map based on a fixed number of channels to obtain the third intermediate feature map; The third intermediate feature map is input into the pooling mapping layer, and the pooling mapping layer compresses the third intermediate feature map to obtain the fourth intermediate feature map. The fourth intermediate feature map is input to the identity mapping layer, which performs an identity mapping on the fourth intermediate feature map and outputs the fourth intermediate feature map as the first spatial feature.

[0016] Optionally, the second feature extraction module is specifically used for: The first spatial feature, the second spatial feature, and the hidden feature from the previous time step are input into the temporal feature extraction model, and then processed sequentially by the feature splicing module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal feature and hidden feature at the current time step.

[0017] Optionally, the second feature extraction module is specifically used for: The first spatial feature and the second spatial feature are input into the feature concatenation module in the temporal feature extraction model. The feature concatenation module concatenates the first spatial feature and the second spatial feature to obtain the concatenated feature. The splicing features and the hidden features of the previous time step are input into the loop module of the temporal feature extraction model. The loop module performs temporal feature extraction based on the splicing features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step. The initial time-series features at the current moment are input into the feature compression module in the time-series feature extraction model. The feature compression module compresses the initial time-series features to obtain the time-series features at the current moment.

[0018] Optionally, the loop module includes a first gated loop layer and a second gated loop layer connected in sequence; The second feature extraction module is specifically used for: The splicing features and the first hidden features of the previous time step are input into the first gated loop layer. The first gated loop layer performs temporal feature extraction on the splicing features and the hidden features to obtain the first temporal features of the current time step and the first hidden features of the current time step. The first temporal feature and the first hidden feature at the current moment are input into the second gated loop layer. The second gated loop layer extracts temporal features from the first temporal feature and the second hidden feature at the previous moment to obtain the initial temporal feature and the second hidden feature at the current moment. The first hidden feature and the second hidden feature at the current moment are used as the hidden feature at the current moment.

[0019] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus, wherein the memory stores program instructions executable by the processor, and when the application runs, the processor communicates with the memory via the bus, and the processor executes the program instructions to perform the steps of the visual perception processing method described in the first aspect above.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which is read and executes the steps of the visual perception processing method described in the first aspect.

[0021] The beneficial effects of this application are: This application provides a visual perception processing method, apparatus, electronic device, and storage medium. After acquiring a first viewpoint image and a second viewpoint image of a target object at the current moment, an improved spatial feature extraction model is used to extract spatial features from the first viewpoint image to obtain first spatial features, and spatial features are extracted from the second viewpoint image to obtain second spatial features. Then, a temporal feature extraction model is used to obtain temporal features and hidden features at the current moment based on the first spatial features, the second spatial features, and the hidden features of the previous moment, so as to make decisions based on temporal features. Compared to existing single-lens applications, this application utilizes a wide-angle vision sensor to obtain first and second viewpoint images, resulting in a larger and more comprehensive physical coverage and viewpoint redundancy for environmental perception. This allows for capturing large-scale scenes at closer range, reducing the probability of field-of-view loss due to mechanical motion. Furthermore, the dual viewpoints provide a perspective difference, laying the foundation for subsequent feature extraction. Subsequently, based on an improved spatial feature extraction model, semantic isomorphic representation of dual-view spatial features is achieved, providing high-fidelity, comparable dual-source input for subsequent temporal feature extraction. Finally, the temporal feature extraction module uses the first and second spatial features as observations and the hidden features from the previous moment as the internal state carrier, considering the continuity of the time dimension, improving the robustness of instantaneous perception, and making the obtained temporal features more accurate, facilitating subsequent decision-making tasks. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is an exemplary scenario diagram provided for an embodiment of this application; Figure 2 A flowchart illustrating a visual perception processing method provided in an embodiment of this application; Figure 3 A flowchart illustrating the second visual perception processing method provided in this application embodiment; Figure 4A flowchart illustrating the third visual perception processing method provided in this application embodiment; Figure 5 A schematic diagram of an apparatus for a visual perception processing method provided in an embodiment of this application; Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0025] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0026] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0027] Optionally, this application can be applied to the field of robotics, and correspondingly, the technical solution of this application can be implemented by electronic devices in the field of robotics. For example, in the field of robotics, the electronic device can be, for instance, a robot or a robot control device.

[0028] Figure 1 An exemplary scenario diagram provided for an embodiment of this application, such as... Figure 1As shown, the scenario includes an electronic device, a left vision sensor mounted on the end of the left robotic arm of the dual-arm robot, and a right vision sensor mounted on the end of the right robotic arm of the dual-arm robot. The left and right vision sensors simultaneously acquire images of the target object within the field of view of the dual-arm robot and send the acquired images to the electronic device. The electronic device uses the visual perception processing method provided in this application embodiment to perform spatial feature extraction and temporal feature extraction, thereby obtaining the feature information of the target object.

[0029] Figure 2 A flowchart illustrating a visual perception processing method provided in an embodiment of this application is shown below. Figure 2 As shown, the subject of this method is the aforementioned electronic device, and the method may include: S101. Obtain the first viewpoint image and the second viewpoint image of the target object at the current moment.

[0030] The first viewpoint image is acquired by a first wide-angle vision sensor, and the second viewpoint image is acquired by a second wide-angle vision sensor. If the first wide-angle vision sensor is a vision sensor mounted on the left arm of the robot, then the second wide-angle vision sensor is a sensor mounted on the right arm of the robot. If the first wide-angle vision sensor is a vision sensor mounted on the right arm of the robot, then the second wide-angle vision sensor is a sensor mounted on the left arm of the robot. The first and second wide-angle vision sensors are, for example, fisheye cameras with a field of view of 177 degrees.

[0031] Optionally, the first wide-angle vision sensor and the second wide-angle vision sensor are configured in a stereo setup approximating the baseline of the human eye. The target object is an entity located within the overlapping field of view of both the first and second wide-angle vision sensors. At any given moment, the electronic device can control the first and second wide-angle vision sensors to simultaneously acquire images of the target object, obtaining a first viewpoint image and a second viewpoint image. The relative spatial relationship between the left and right robotic arms and the target object can be simultaneously obtained through the first and second wide-angle vision sensors.

[0032] S102. Input the first viewpoint image and the second viewpoint image into the improved spatial feature extraction model respectively. The improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain the first spatial feature, and extracts spatial features from the second viewpoint image to obtain the second spatial feature.

[0033] In this design, the first spatial feature and the second spatial feature have the same spatial dimension. For example, if the first spatial feature is a 1×1×D dimensional feature, the second spatial feature is also a 1×1×D dimensional feature, where D is, for example, 128 dimensions. The first spatial feature and the second spatial feature constitute a semantically aligned common representation space. That is, in this common representation space, any dimension does not refer to a specific attribute of a particular visual sensor or viewpoint, but rather is a spatial feature shared by both viewpoints with respect to the target object.

[0034] For example, spatial features can refer to the shape features, texture features, and local set features of the target object at the current moment. That is to say, spatial features are used to indicate the static structure of the target object at the current moment.

[0035] Optionally, the first viewpoint image and the second viewpoint image are respectively input into two logically independent but physically parameter-bound feature extraction paths in the spatial feature extraction model. This allows the spatial feature extraction model to extract spatial features from the first viewpoint image at the current moment, obtaining image information features from one perspective at that moment, and to extract spatial features from the second viewpoint image at the current moment, obtaining image information features from another perspective at that moment. The process of extracting spatial features from the first and second viewpoint images can be performed simultaneously and independently. The feature extraction model adopts a multi-level cascaded structure.

[0036] S103. Input both the first spatial features and the second spatial features into the temporal feature extraction model. The temporal feature extraction model obtains the temporal features and hidden features of the current moment based on the first spatial features, the second spatial features, and the hidden features of the previous moment, so as to make decisions based on the temporal features.

[0037] The hidden features of the previous time step include temporal hidden features from the previous time step and other time steps prior to it, such as features like historical consistency constraints and behavioral inertia. These features are used to correct single-frame perception errors and predict short-term motion trends. The hidden features of the previous time step can be stored in the database of the temporal feature extraction model and can be retrieved from this database during temporal feature extraction.

[0038] For example, hidden features can be features such as the target object's motion trend (e.g., uniform speed, acceleration, periodic oscillation), interaction intent (e.g., approaching, moving away, remaining still), and environmental context (e.g., workbench stability). In other words, hidden features are used to indicate the target object's dynamic behavior across frames, containing a variety of historical information from the beginning to the current moment.

[0039] Optionally, although the first and second spatial features originate from different sensor perspectives, after spatial feature extraction, they are mapped to the same dimensional common representation space. However, both the first and second spatial features only extract image information features at a single moment, while the temporal feature extraction model extracts these single-moment image information features in a chronological sequence. Addressing the temporal nature of visual information, this temporal feature extraction model obtains the temporal features of the current moment based on the input first and second spatial features and the hidden features of the previous moment pre-stored in the database. That is, using the first and second spatial features as observations and the hidden features of the previous moment as the internal state carrier, their synergistic effect ensures that the obtained temporal features of the current moment not only respond to the visual input at the current moment but also reflect the understanding and reasoning regarding dynamic behavior.

[0040] Optionally, after obtaining the temporal features and hidden features at the current moment, the temporal features are output to the subsequent decision model. In addition, the hidden features at the current moment are stored in the database of the temporal feature extraction model to facilitate the extraction of temporal features at subsequent moments.

[0041] In this embodiment, after acquiring the first viewpoint image and the second viewpoint image of the target object at the current moment, the improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain the first spatial feature, and extracts spatial features from the second viewpoint image to obtain the second spatial feature. Then, the temporal feature extraction model obtains the temporal feature and hidden feature at the current moment based on the first spatial feature, the second spatial feature, and the hidden feature of the previous moment, so as to make a decision based on the temporal feature. Compared to existing single-lens applications, this application utilizes a wide-angle vision sensor to obtain first and second viewpoint images, resulting in a larger and more comprehensive physical coverage and viewpoint redundancy for environmental perception. This allows for capturing large-scale scenes at closer range, reducing the probability of field-of-view loss due to mechanical motion. Furthermore, the dual viewpoints provide a perspective difference, laying the foundation for subsequent feature extraction. Subsequently, based on an improved spatial feature extraction model, semantic isomorphic representation of dual-view spatial features is achieved, providing high-fidelity, comparable dual-source input for subsequent temporal feature extraction. Finally, the temporal feature extraction module uses the first and second spatial features as observations and the hidden features from the previous moment as the internal state carrier, considering the continuity of the time dimension, improving the robustness of instantaneous perception, and making the obtained temporal features more accurate, facilitating subsequent decision-making tasks.

[0042] Optionally, the improved spatial feature extraction model includes an initial convolutional pooling layer, a standard intermediate layer, a channel-locking layer, a pooling mapping layer, and an identity mapping layer connected in sequence, wherein the channel-locking layer keeps the number of channels unchanged when the feature map is downsampled.

[0043] Optionally, in S102 above, the improved spatial feature extraction model is used to extract spatial features from the first viewpoint image to obtain first spatial features, and the improved spatial feature extraction model is used to extract spatial features from the second viewpoint image to obtain second spatial features. This may include: Specifically, the first viewpoint image is input into the improved spatial feature extraction model, and spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locked layer, pooling mapping layer and identity mapping layer in the spatial feature extraction model to obtain the first spatial features.

[0044] Figure 3 A flowchart illustrating the second visual perception processing method provided in this application embodiment is shown below. Figure 3 As shown, the first viewpoint image is input into the improved spatial feature extraction model, and spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locking layer, pooling mapping layer, and identity mapping layer in the spatial feature extraction model to obtain the first spatial features, which may include: S201. Input the first viewpoint image into the initial convolutional pooling layer. The initial convolutional pooling layer performs convolution and downsampling processing on the first viewpoint image to obtain the first intermediate feature map.

[0045] Optionally, the initial convolutional pooling layer may include the initial convolutional layer and pooling layer of ResNet-18, and the parameters of the initial convolutional layer may be, for example, 7. 7 Conv, Stride 2, meaning the kernel size is 7. 7. With a stride of 2, the initial convolutional layer is used to transform the input image size from H×W to... The pooling layer transforms the number of channels to the first number of channels; for example, the parameters of this pooling layer can be 3. 3 MaxPool, Stride 2, the pooling window of the pooling layer is 3. 3. With a stride of 2, the pooling layer converts the feature map output from the initial convolutional layer into... Transform into And keep the number of channels unchanged.

[0046] For example, if the first viewpoint image is (3, 128, 128), where 3 is the number of RGB channels, 128 is the image height of the first viewpoint image, and 128 is the image width of the first viewpoint image, after the initial convolutional layer processes the size and number of channels of the first viewpoint image, the size of the output feature map is 64. 64, the number of channels is 64, then 64 64 is then input to the pooling layer, and after pooling processing, 32 is output. The first intermediate feature map is 32, and the number of output channels remains unchanged at 64.

[0047] S202. Input the first intermediate feature map into the standard intermediate layer, and the standard intermediate layer performs downsampling processing on the first intermediate feature map to obtain the second intermediate feature map.

[0048] Optionally, the standard intermediate layer may include two standard residual blocks, such as layer1 and layer2, wherein layer1 is not downsampled and maintains the resolution of the first intermediate feature map, that is, the resolution of the feature map output by layer1 is still... And the number of channels is still the first number of channels; layer2 downsamples the feature map output by layer to And increase the number of channels to the second number of channels.

[0049] For example, layer 1 outputs 32. The feature map is 32 pixels, and the number of channels is still 64; the stride 2 of layer 2 means that the feature map output from layer 1 is downsampled to 16 pixels after passing through layer 2. 16. The number of channels is increased to 128. Therefore, the size of the resulting second intermediate feature map is 16. 16, with 128 channels.

[0050] S203. The second intermediate feature map is input to the channel locking layer, and the channel locking layer performs downsampling processing on the second intermediate feature map based on a fixed number of channels to obtain the third intermediate feature map.

[0051] Optionally, the channel-locked layer includes layer 3 and layer 4, where the number of channels in layer 3 and layer 4 remains unchanged, and they are in a locked state. Layer 3 downsamples the second intermediate feature map to... And keep the number of channels unchanged; layer4 downsamples the feature map output by layer3 to And keep the number of channels unchanged.

[0052] For example, the second intermediate feature map is downsampled via layer 3 to obtain 8 The feature map of 8, the 8 The feature map of 8 retains 128 channels; then 8 The feature map of 8 is input to layer 4, and layer 8... The feature map of 8 is downsampled to obtain 4. The feature map of 4, and the 4 The number of channels in the 4-channel feature map remains unchanged, meaning the size of the resulting third intermediate feature map is 4. 4. Keep the number of channels at 128.

[0053] S204. Input the third intermediate feature map into the pooling mapping layer, and the pooling mapping layer compresses the third intermediate feature map to obtain the fourth intermediate feature map.

[0054] Optionally, the pooling window size of this pooling mapping layer is 4. 4. The step size is 1, which is 4. 4. MaxPool, Stride 1. The pooling mapping layer can fix the size of the third intermediate feature map to 1. 1, and keep the number of channels unchanged.

[0055] For example, the pooling mapping layer will 4 The third intermediate feature map of 4 is compressed through this pooling mapping layer to obtain 1. The fourth intermediate feature map of 1 has 128 channels.

[0056] S205. Input the fourth intermediate feature map into the identity mapping layer, perform identity mapping on the fourth intermediate feature map by the identity mapping layer, and output the fourth intermediate feature map as the first spatial feature.

[0057] Optionally, the identity mapping layer directly outputs the fourth intermediate feature map as the first spatial feature, and the number of channels of the output first spatial feature is the number of channels of the second spatial feature. For example, the number of channels of the first spatial feature is 128.

[0058] Traditional ResNet18 connects a fully connected layer after the pooling layer. In this embodiment, the fully connected layer is improved to an identity mapping layer, which flattens the fourth intermediate feature map output by the pooling mapping layer or directly outputs it as the first spatial feature.

[0059] In this embodiment, the number of channels is locked by a channel locking layer, which reduces the feature dimension and the amount of computation. At the same time, it avoids overfitting of high-dimensional features in small-scale space. The output of each channel is compressed by a pooling mapping layer, and the feature map output by the pooling mapping layer is avoided by an identity mapping layer. The output of the pooling mapping layer is directly used as the output of the spatial feature advance model, avoiding the position sensitivity caused by the fully connected layer, keeping the spatial aggregation of features purer, and reducing the introduction of other parameters.

[0060] It is worth noting that the feature extraction process of the second spatial features is similar to the process of extracting the first spatial features in steps S201 to S205, and will not be elaborated here.

[0061] Optionally, in S103 above, the temporal feature extraction model obtains the temporal features and hidden features of the current moment based on the first spatial features, the second spatial features, and the hidden features of the previous moment. This may include: Specifically, the first spatial features, the second spatial features, and the hidden features from the previous time step are input into the temporal feature extraction model, and then processed sequentially by the feature splicing module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal features and hidden features at the current time step.

[0062] Figure 4 A flowchart illustrating the third visual perception processing method provided in this application embodiment is shown below. Figure 4 As shown, the above processes are sequentially handled by the feature concatenation module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal features and hidden features at the current time, which may include: S301. Input the first spatial feature and the second spatial feature into the feature concatenation module in the temporal feature extraction model. The feature concatenation module concatenates the first spatial feature and the second spatial feature to obtain the concatenated feature.

[0063] The feature stitching module is a cascaded operation layer along the channel dimension. It receives a first spatial feature and a second spatial feature of the same size, performs stitching along the channel dimension, and outputs the stitched feature. This stitched feature is then input into subsequent loop modules.

[0064] Optionally, if the feature dimension of the first spatial feature is D1 and the feature dimension of the second spatial feature is D2, the feature stitching module stitches the feature dimensions of the first spatial feature and the second spatial feature together, resulting in a stitched feature with a feature dimension of D1 + D2. The resulting stitched feature contains the complete spatial features of binocular vision, preserving the disparity information between the left and right viewpoints. Binocular vision relies heavily on the difference information between the left and right viewpoints; for example, depth estimation depends on this difference information.

[0065] For example, if the number of channels of the first spatial feature is 128 and the number of channels of the second spatial feature is also 128, then the resulting spliced ​​feature has a dimension of 256.

[0066] S302. Input the spliced ​​features and the hidden features of the previous time step into the loop module in the temporal feature extraction model. The loop module extracts temporal features based on the spliced ​​features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step.

[0067] The recurrent module is a spatial-temporal coupled gated recurrent unit. This module learns the weight matrix and autonomously learns the correlation between left and right viewpoints and the temporal dependence of visual information based on the splicing features and the hidden features from the previous time step, achieving deep fusion of spatial and temporal information. For example, the initial temporal features obtained at the current time step are 128-dimensional features, and the hidden features at the current time step are also 128-dimensional features.

[0068] Optionally, if at the current time t=0, there are no hidden features from the previous time, then the hidden features from the previous time are initialized to 0. This provides a clean initial state for subsequent time series modeling and avoids introducing bias information.

[0069] S303. Input the initial time series features at the current time into the feature compression module in the time series feature extraction model. The feature compression module compresses the initial time series features to obtain the time series features at the current time.

[0070] Optionally, after obtaining the initial time-series features for the current moment, these features may contain redundant channel responses and low signal-to-noise ratio spatiotemporal noise, such as background jitter and sudden illumination disturbances. Therefore, these initial time-series features are input into a compression module for compression processing to obtain the time-series features for the current moment. For example, the obtained time-series features for the current moment may be 32-dimensional. Through the compression processing of the compression module, the initial time-series features can be compressed, reducing the computational burden on downstream tasks.

[0071] In this embodiment, the spatial relationship prior is implemented to explicitly guide the evolution of the temporal state, significantly reducing the divergence of the hidden state of the continuous loop module in the scene of violent movement. Furthermore, the initial temporal features are compressed, which can remove temporal disturbance components that are irrelevant to the downstream task while retaining discriminative features, thereby improving the feature signal-to-noise ratio and making the results of the downstream task more accurate.

[0072] Optionally, in S302 above, the loop module extracts temporal features based on the splicing features and the hidden features from the previous time step to obtain the initial temporal features and the hidden features of the current time step, which may include: The loop module includes a first gated recurrent unit (GRU) and a second gated recurrent unit connected in sequence.

[0073] Specifically, the spliced ​​features and the hidden features from the previous time step are input into the first gated recurrent layer, which then performs temporal feature extraction on the spliced ​​features and the hidden features to obtain the first temporal features and the first hidden features.

[0074] The first gated recurrent layer can be an update gate, which determines how much historical information needs to be retained in the current time step and which needs to be updated. This first gated recurrent layer focuses on the local temporal structure and extracts short-term dependencies. Based on the stitching features and the hidden features from the previous time step, the first gated recurrent layer performs preliminary temporal modeling, learning the temporal changes of the binocular image at the current time step, such as object motion speed, optical flow visual changes, and disparity change rate, to obtain the first temporal feature and the first hidden feature at the current time step.

[0075] The first temporal feature at the current time step and the second hidden feature at the previous time step are input into the second gated recurrent layer. The second gated recurrent layer extracts temporal features from the first temporal feature and the first hidden feature, learns high-level behavioral patterns, and obtains the initial temporal feature and the second hidden feature at the current time step. The first hidden feature and the second hidden feature at the current time step are then used as the hidden features at the current time step.

[0076] The second gated recurrent layer can act as a reset gate, determining how much historical information needs to be ignored. This second gated recurrent layer focuses on the global temporal structure. It performs high-level abstraction on the first temporal features output by the first gated recurrent layer and the second hidden features from the previous time step, capturing more complex temporal dependencies, such as driving intention, action type, and event prediction information, to obtain the initial temporal features and the second hidden features of the current time step. These initial and second hidden features are then used as the hidden features of the current time step. The initial temporal features describe the temporal change features obtained from multi-frame image fusion; they are externally transmitted features, directly output to the next model for final decision-making. The hidden features of the current time step refer to the memory carrier of all historical information from the beginning to the current time step; they are not directly output but internally transmitted to the next time step to maintain memory and resolve long-term dependencies.

[0077] Figure 5 A schematic diagram of an apparatus for a visual perception processing method provided in an embodiment of this application is shown below. Figure 5 As shown, the device includes: The acquisition module 401 is used to acquire a first viewpoint image and a second viewpoint image of the target object at the current moment. The first viewpoint image is acquired by a first wide-angle vision sensor, and the second viewpoint image is acquired by a second wide-angle vision sensor. The first feature extraction module 402 is used to input the first viewpoint image and the second viewpoint image into the improved spatial feature extraction model respectively. The improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain a first spatial feature, and extracts spatial features from the second viewpoint image to obtain a second spatial feature. The first spatial feature and the second spatial feature have the same dimension. The second feature extraction module 403 is used to input both the first spatial feature and the second spatial feature into the temporal feature extraction model. The temporal feature extraction module obtains the temporal feature and hidden feature of the current moment based on the first spatial feature, the second spatial feature and the hidden feature of the previous moment of the current moment, so as to make a decision based on the temporal feature and hidden feature.

[0078] Optionally, the improved spatial feature extraction model includes an initial convolutional pooling layer, a standard intermediate layer, a channel-locking layer, a pooling mapping layer, and an identity mapping layer connected in sequence, wherein the channel-locking layer keeps the number of channels unchanged when the feature map is downsampled.

[0079] Optionally, the first feature extraction module 402 is specifically used for: The first viewpoint image is input into the improved spatial feature extraction model, and spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locked layer, pooling mapping layer and identity mapping layer in the spatial feature extraction model to obtain the first spatial feature.

[0080] Optionally, the first feature extraction module 402 is specifically used for: The first viewpoint image is input into the initial convolutional pooling layer, and the initial convolutional pooling layer performs convolution and downsampling processing on the first viewpoint image to obtain the first intermediate feature map; The first intermediate feature map is input into the standard intermediate layer, and the standard intermediate layer performs downsampling processing on the first intermediate feature map to obtain the second intermediate feature map. The second intermediate feature map is input to the channel locking layer, and the channel locking layer performs downsampling processing on the second intermediate feature map based on a fixed number of channels to obtain the third intermediate feature map; The third intermediate feature map is input into the pooling mapping layer, and the pooling mapping layer compresses the third intermediate feature map to obtain the fourth intermediate feature map. The fourth intermediate feature map is input to the identity mapping layer, which performs an identity mapping on the fourth intermediate feature map and outputs the fourth intermediate feature map as the first spatial feature.

[0081] Optionally, the second feature extraction module 403 is specifically used for: The first spatial feature, the second spatial feature, and the hidden feature from the previous time step are input into the temporal feature extraction model, and then processed sequentially by the feature splicing module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal feature and hidden feature at the current time step.

[0082] Optionally, the second feature extraction module 403 is specifically used for: The first spatial feature and the second spatial feature are input into the feature concatenation module in the temporal feature extraction model. The feature concatenation module concatenates the first spatial feature and the second spatial feature to obtain the concatenated feature. The splicing features and the hidden features of the previous time step are input into the loop module of the temporal feature extraction model. The loop module performs temporal feature extraction based on the splicing features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step. The initial time-series features at the current moment are input into the feature compression module in the time-series feature extraction model. The feature compression module compresses the initial time-series features to obtain the time-series features at the current moment.

[0083] Optionally, the loop module includes a first gated loop layer and a second gated loop layer connected in sequence; The second feature extraction module 403 is specifically used for: The splicing features and the first hidden features of the previous time step are input into the first gated loop layer. The first gated loop layer performs temporal feature extraction on the splicing features and the hidden features to obtain the first temporal features of the current time step and the first hidden features of the current time step. The first temporal feature and the first hidden feature at the current moment are input into the second gated loop layer. The second gated loop layer extracts temporal features from the first temporal feature and the second hidden feature at the previous moment to obtain the initial temporal feature and the second hidden feature at the current moment. The first hidden feature and the second hidden feature at the current moment are used as the hidden feature at the current moment.

[0084] Figure 6 This is a structural block diagram of an electronic device 500 provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device may include: processor 501 and memory 502.

[0085] Optionally, a bus 503 may also be included, wherein the memory 502 is used to store machine-readable instructions executable by the processor 501. When the electronic device 500 is running, the processor 501 and the memory 502 communicate via the bus 503, and the processor 501 executes the machine-readable instructions to perform the method steps in the above method embodiments.

[0086] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the method steps described in the above-described visual perception processing method embodiments.

[0087] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0089] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A visual perception processing method, characterized in that, The method includes: Acquire a first viewpoint image and a second viewpoint image of the target object at the current moment. The first viewpoint image is acquired by a first wide-angle vision sensor, and the second viewpoint image is acquired by a second wide-angle vision sensor. The first viewpoint image and the second viewpoint image are respectively input into the improved spatial feature extraction model. The improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain the first spatial feature, and extracts spatial features from the second viewpoint image to obtain the second spatial feature. The first spatial feature and the second spatial feature have the same spatial dimension. Both the first spatial feature and the second spatial feature are input into the temporal feature extraction model. The temporal feature extraction model obtains the temporal features and hidden features of the current moment based on the first spatial feature, the second spatial feature, and the hidden features of the previous moment of the current moment, so as to make decisions based on the temporal features.

2. The visual perception processing method according to claim 1, characterized in that, The improved spatial feature extraction model includes an initial convolutional pooling layer, a standard intermediate layer, a channel-locking layer, a pooling mapping layer, and an identity mapping layer connected in sequence. The channel-locking layer keeps the number of channels unchanged when the feature map is downsampled.

3. The visual perception processing method according to claim 2, characterized in that, The step of extracting spatial features from the first viewpoint image using the improved spatial feature extraction model to obtain the first spatial features includes: The first viewpoint image is input into the improved spatial feature extraction model, and spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locked layer, pooling mapping layer and identity mapping layer in the spatial feature extraction model to obtain the first spatial feature.

4. The visual perception processing method according to claim 3, characterized in that, The spatial features are extracted sequentially through the initial convolutional pooling layer, standard intermediate layer, channel-locking layer, pooling mapping layer, and identity mapping layer in the spatial feature extraction model to obtain the first spatial feature, including: The first viewpoint image is input into the initial convolutional pooling layer, and the initial convolutional pooling layer performs convolution and downsampling processing on the first viewpoint image to obtain the first intermediate feature map; The first intermediate feature map is input into the standard intermediate layer, and the standard intermediate layer performs downsampling processing on the first intermediate feature map to obtain the second intermediate feature map. The second intermediate feature map is input to the channel locking layer, and the channel locking layer performs downsampling processing on the second intermediate feature map based on a fixed number of channels to obtain the third intermediate feature map; The third intermediate feature map is input into the pooling mapping layer, and the pooling mapping layer compresses the third intermediate feature map to obtain the fourth intermediate feature map. The fourth intermediate feature map is input to the identity mapping layer, which performs an identity mapping on the fourth intermediate feature map and outputs the fourth intermediate feature map as the first spatial feature.

5. The visual perception processing method according to claim 1, characterized in that, The step of obtaining the temporal features and hidden features of the current moment by the temporal feature extraction model based on the first spatial features, the second spatial features, and the hidden features of the previous moment of the current moment includes: The first spatial feature, the second spatial feature, and the hidden feature from the previous time step are input into the temporal feature extraction model, and then processed sequentially by the feature splicing module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal feature and hidden feature at the current time step.

6. The visual perception processing method according to claim 5, characterized in that, The features are processed sequentially through the feature concatenation module, the loop processing module, and the compression module in the temporal feature extraction model to obtain the temporal features and hidden features at the current time, including: The first spatial feature and the second spatial feature are input into the feature concatenation module in the temporal feature extraction model. The feature concatenation module concatenates the first spatial feature and the second spatial feature to obtain the concatenated feature. The splicing features and the hidden features of the previous time step are input into the loop module of the temporal feature extraction model. The loop module performs temporal feature extraction based on the splicing features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step. The initial time-series features at the current moment are input into the feature compression module in the time-series feature extraction model. The feature compression module compresses the initial time-series features to obtain the time-series features at the current moment.

7. The visual perception processing method according to claim 6, characterized in that, The loop module includes a first gated loop layer and a second gated loop layer connected in sequence; The step of extracting temporal features by the loop module based on the splicing features and the hidden features of the previous time step to obtain the initial temporal features and the hidden features of the current time step includes: The splicing features and the first hidden features of the previous time step are input into the first gated loop layer. The first gated loop layer performs temporal feature extraction on the splicing features and the hidden features to obtain the first temporal features of the current time step and the first hidden features of the current time step. The first temporal feature and the first hidden feature at the current moment are input into the second gated loop layer. The second gated loop layer extracts temporal features from the first temporal feature and the second hidden feature at the previous moment to obtain the initial temporal feature and the second hidden feature at the current moment. The first hidden feature and the second hidden feature at the current moment are used as the hidden feature at the current moment.

8. A visual perception processing device, characterized in that, include: The acquisition module is used to acquire a first viewpoint image and a second viewpoint image of the target object at the current moment. The first viewpoint image is acquired by a first wide-angle vision sensor, and the second viewpoint image is acquired by a second wide-angle vision sensor. The first feature extraction module is used to input the first viewpoint image and the second viewpoint image into the improved spatial feature extraction model, and the improved spatial feature extraction model extracts spatial features from the first viewpoint image to obtain a first spatial feature, and extracts spatial features from the second viewpoint image to obtain a second spatial feature, wherein the first spatial feature and the second spatial feature have the same dimension. The second feature extraction module is used to input both the first spatial feature and the second spatial feature into the temporal feature extraction model. The temporal feature extraction module obtains the temporal feature and hidden feature of the current moment based on the first spatial feature, the second spatial feature, and the hidden feature of the previous moment of the current moment, so as to make a decision based on the temporal feature and hidden feature.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program executable by the processor, and the processor executes the computer program to implement the steps of the visual perception processing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the visual perception processing method as described in any one of claims 1-7.

Images