3D object detection method and device based on inter-modal implicit spatial enhancement and spatio-temporal interaction encoding

Abstract

The method and device for 3D target detection based on inter-modal implicit space enhancement and space-time interactive coding, the method comprises the following steps: acquiring training data and constructing a 3D target detection architecture; acquiring RGB image features and point cloud features respectively through a feature extraction network, and inputting the features into an implicit space enhancement fusion coding network to generate bird's eye view image features containing rich geometric information; fusing the image bird's eye view features and the point cloud bird's eye view features in the spatial and temporal dimensions through a space-time interactive feature fusion network to obtain final multi-modal fusion features; and sending the fusion features into a classification head to realize target detection and identification. The application fully excavates the complementary characteristics of LiDAR point cloud and RGB image to perform high-precision target detection.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional target detection, and particularly to a 3D target detection method and device based on inter-modal implicit space enhancement and spatio-temporal interaction encoding. BACKGROUND

[0002] In recent years, 3D target detection technology has attracted great attention in the fields of robots and autonomous driving. Traditional three-dimensional target detection methods usually rely on single modal data, such as LiDAR point clouds or RGB images. These methods perform well in specific scenarios, but have limitations in complex and dynamic environments. For example, LiDAR point clouds perform well in outdoor environments, but have difficulty detecting small objects in occlusion and complex backgrounds; while RGB images provide rich texture information, but perform poorly in depth perception and adverse weather conditions. Therefore, effectively integrating multi-modal data to achieve accurate perception is crucial in autonomous driving, and multi-modal fusion is a key method to solve this problem.

[0003] Currently, multi-modal fusion models mainly fall into two methods: perspective transformation and splicing fusion based on cross-modal interaction fusion. These methods aim to align LiDAR point cloud and RGB image features, achieve effective feature fusion, and use fused features for bounding box and class prediction. However, existing methods face many challenges in 3D target detection.

[0004] Firstly, RGB images lack depth information and usually rely on explicit depth estimation methods to align LiDAR data, but errors are easily produced in complex environments, leading to inaccurate conversion of image features to bird's eye view (BEV) space, affecting the quality of fused features and detection results. Secondly, LiDAR point clouds provide accurate three-dimensional spatial information, but lack color and texture, limiting target classification and recognition capabilities, and point cloud sparsity limits small object detection. Finally, existing fusion models often use simple splicing, failing to fully exploit the complementary information of LiDAR and RGB images, limiting the expressiveness of multi-modal features, and facing sensor failure and data damage problems in practical applications, affecting their reliability. SUMMARY

[0005] The present application overcomes the above-mentioned shortcomings of the prior art and provides a 3D target detection method and device based on inter-modal implicit space enhancement and spatio-temporal interaction encoding.

[0006] The application provides a 3D target detection method based on inter-modal implicit space enhancement and space-time interactive coding.

[0007] In a first aspect, the application provides a 3D target detection method based on inter-modal implicit space enhancement and space-time interactive coding, which comprises the following steps:

[0008] S1. Obtain a one-to-one corresponding multi-view RGB image data and LiDAR point cloud data as a training sample, and construct a 3D target detection architecture, wherein the 3D target detection architecture comprises a feature extraction network, an implicit space enhancement fusion coding network, a space-time interactive feature fusion network, and a classification head.

[0009] S2. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data in the training sample into the feature extraction network for feature extraction to obtain RGB image features and LiDAR point cloud features.

[0010] S3. The implicit space enhancement fusion coding network comprises a preliminary interaction unit and a fine interaction unit, wherein in the preliminary interaction unit, the LiDAR point cloud data is projected into the RGB image coordinate system by using the intrinsic matrix and extrinsic matrix of the camera, and the spatial depth information of the RGB image features is dynamically enhanced by using a variable cross-attention mechanism, and in the fine interaction unit, the original RGB image features and the enhanced RGB image features are fused by using a residual network to obtain the RGB image features in the bird's eye view.

[0011] S4. Input the LiDAR point cloud features and the RGB image features in the bird's eye view into the space-time interactive feature fusion network, wherein the space-time interactive feature fusion network comprises a spatial fusion unit and a temporal fusion unit, the LiDAR point cloud features and the RGB image features in the bird's eye view are applied to a deformable cross-attention to obtain preliminary bird's eye features in the spatial fusion unit, and the preliminary bird's eye features are aggregated by using a deformable self-attention mechanism to obtain final fusion features in the temporal fusion unit.

[0012] S5. Input the final fusion features into the classification head to obtain a 3D target detection result, construct a loss function based on the detection result, save the current parameters of the 3D target detection architecture when the loss function meets a set condition, and obtain a 3D target detection model.

[0013] S6. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data into the trained 3D target detection model to obtain a target detection result.

[0014] The preliminary interaction unit in step S3 comprises a projection operation layer and a spatial coding layer. In the projection operation layer, the LiDAR point cloud data is projected onto the RGB image feature by using the intrinsic matrix and the extrinsic matrix of the camera to establish a coordinate system correspondence; in the spatial coding layer, the enhanced image feature is obtained by using the accurate spatial information of the LiDAR data and combining the variable cross-attention mechanism.

[0015] The fine interaction unit in step S3 comprises two residual modules, which fuse the original RGB image feature and the enhanced RGB image feature by using a residual network to obtain the RGB image feature in the bird's eye view.

[0016] The spatial fusion unit in step S4 comprises a query generation and position coding layer, a deformable cross-attention layer and a residual layer. The query generation and position coding layer generates a query vector corresponding to a specific spatial position in the ROI grid and performs position coding on the query vector to capture the relative spatial information of the position. The deformable cross-attention layer uses the query vector to fuse the point cloud bird's eye feature and the image bird's eye feature. Finally, the residual layer performs residual connection to obtain the preliminary bird's eye fusion feature.

[0017] The temporal fusion unit in step S4 comprises an initialization layer, a self-attention update layer, a feedforward network layer and a fusion layer. The preliminary bird's eye fusion feature is initialized in the initialization layer by using a self-attention mechanism to initialize a query vector; then, in the self-attention update layer, a deformable attention mechanism is used to fuse the encoded features of the historical frames into the query vector to update the query vector. The updated query vector is processed by the feedforward network layer to further enrich its feature representation, and finally the temporal information is fused by the fusion layer to obtain the final bird's eye fusion feature.

[0018] The second aspect of the present application relates to a 3D target detection device based on inter-modal implicit spatial enhancement and spatio-temporal interaction coding, comprising a memory and one or more processors, the memory storing executable code, and the one or more processors executing the executable code to implement the 3D target detection method based on inter-modal implicit spatial enhancement and spatio-temporal interaction coding of the present application.

[0019] The third aspect of the present application relates to a computer readable storage medium having a program stored thereon, which is executed by a processor to implement the 3D target detection method based on inter-modal implicit spatial enhancement and spatio-temporal interaction coding of the present application.

[0020] The advantages of the present application are as follows:

[0021] The present application uses the spatial information of LiDAR data for implicit depth estimation through implicit spatial enhancement fusion coding and variable cross-attention mechanism, improves the expression ability of RGB image features, and dynamically adjusts the interaction intensity between RGB image features and LiDAR point cloud features. In addition, the present application also introduces a spatio-temporal interactive feature fusion method, which captures the mutual relationship between modalities and aggregates semantic information in the spatial and temporal dimensions through deformable self-attention and deformable cross-attention mechanism, and improves the detection accuracy and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application. In the drawings:

[0023] Figure 1 is a flowchart of a 3D target detection method based on inter-modal implicit spatial enhancement and spatio-temporal interactive coding according to an embodiment of the present application;

[0024] Figure 2 is a whole structure diagram of a 3D target detection model according to an embodiment of the present application;

[0025] Figure 3 is a structure diagram of an implicit spatial enhancement fusion coding network according to an embodiment of the present application;

[0026] Figure 4 is a structure diagram of a spatio-temporal interactive feature fusion network according to an embodiment of the present application; DETAILED DESCRIPTION

[0027] The exemplary embodiments will be described in detail herein below with reference to the drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of the description. Instead, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of the description as detailed in the appended claims.

[0028] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the specification in other embodiments. In some other embodiments, the steps included in the method thereof can be more or less than described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; while multiple steps described in the specification can also be combined into a single step for description in other embodiments.

[0029] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects and advantages of the present application more clear.

[0030] Embodiment one

[0031] The present example proposes a 3D target detection method based on inter-modal implicit space enhancement and space-time interactive encoding. Through the implicit space enhancement fusion encoding module, the spatial information of LiDAR and the attention mechanism are used to enhance the geometric details of the RGB image features, and the LiDAR point cloud is projected to the image plane through the camera internal and external parameter matrix for feature correspondence. In addition, a space-time interactive feature fusion is adopted, which uses a deformable attention mechanism to capture the inter-modal feature relationship in the spatial and temporal dimensions, aggregates multi-modal semantic information, generates enhanced bird's eye view features, and fuses historical frame information to improve detection accuracy. Specifically, referring to Figure 1 With Figure 2 , the method comprises:

[0032] S1. Obtain a one-to-one corresponding multi-view RGB image data and LiDAR point cloud data as a training sample, and construct a 3D target detection architecture, which comprises a feature extraction network, an implicit space enhancement fusion encoding network, a space-time interactive feature fusion network and a classification head;

[0033] S2. Input a set of corresponding multi-view RGB image data and LiDAR point cloud data in the training sample into the feature extraction network for feature extraction to obtain RGB image features and LiDAR point cloud features;

[0034] S3. The implicit space enhancement fusion encoding network comprises a preliminary interaction unit and a fine interaction unit. In the preliminary interaction unit, the LiDAR point cloud data is projected into the RGB image coordinate system by using the internal and external parameter matrices of the camera, and the spatial depth information of the RGB image features is dynamically enhanced by using the variable cross-attention mechanism. In the fine interaction unit, the original RGB image features and the enhanced RGB image features are fused by using the residual network to obtain the RGB image features under the bird's eye view angle;

[0035] S4. respectively input the LiDAR point cloud feature and the RGB image feature under the bird's eye view into a space-time interactive feature fusion network, the space-time interactive feature fusion network comprising a space fusion unit and a time fusion unit, the space fusion unit applying a deformable cross attention to the LiDAR point cloud feature and the RGB image feature under the bird's eye view to obtain a preliminary bird's eye view feature, and the time fusion unit using a deformable self-attention mechanism to aggregate the preliminary bird's eye view feature to obtain a final fusion feature.

[0036] S5. input the final fusion feature into a classification head to obtain a 3D target detection result, construct a loss function based on the detection result, save the current parameters of the 3D target detection architecture when the loss function meets a set condition, and obtain a 3D target detection model.

[0037] S6. input a set of corresponding multi-view RGB image data and LiDAR point cloud data into the trained 3D target detection model to obtain a target detection result.

[0038] In the present scheme, the multi-view RGB image is a plurality of images taken by a plurality of camera devices at the same position, which collectively reflect the complete situation of all targets in the region. The LiDAR point cloud data is a set of spatial points generated by scanning a three-dimensional laser radar device, each point containing three-dimensional coordinate information of the target.

[0039] In the present scheme, the feature extraction network is composed of an RGB feature extraction unit and a LiDAR point cloud feature extraction unit, the RGB feature extraction unit being used for feature extraction of the multi-view RGB image, and the LiDAR point cloud feature extraction unit being used for feature extraction of the LiDAR point cloud data.

[0040] Specifically, the RGB feature extraction unit and the LiDAR point cloud feature extraction unit in the present scheme can both use any convolutional neural network architecture for feature extraction.

[0041] In the present scheme, the structure of the implicit space-enhanced fusion encoding network is as shown in Figure 3 The implicit space-enhanced fusion encoding network is described as follows:

[0042] In the step of "the preliminary interaction unit projects the LiDAR point cloud data into the RGB image coordinate system by using the intrinsic matrix and the extrinsic matrix of the camera, and dynamically enhances the spatial depth information of the RGB image features through the variable cross-attention mechanism", the preliminary interaction unit includes a projection operation layer, a spatial coding layer and a feature fusion layer. In the projection operation layer, the LiDAR point cloud data is projected onto the RGB image features by using the intrinsic matrix and the extrinsic matrix of the camera, and the coordinate system correspondence is established; in the spatial coding layer, the enhanced image features are obtained through operation by using the accurate spatial information of the LiDAR data combined with the variable cross-attention mechanism.

[0043] Specifically, the LiDAR point cloud data is mapped to the two-dimensional image coordinate system where the image features are located by using the intrinsic matrix and the extrinsic matrix of the camera, and the spatial correspondence between the coordinate systems is established. The specific formula is as follows:

[0044]

[0045] Where K represents the intrinsic matrix of the camera, [R|t] represents the extrinsic matrix from the LiDAR coordinate system to the camera coordinate system, {(x, y, z)} represents the laser point, and {(u, v)} represents the point converted to the camera coordinate system.

[0046] Further, the feature point set of the LiDAR point cloud is defined as P LiDAR ={p1, p2, … p N}, and the variable cross-attention mechanism is applied between the LiDAR point p i ={x, y, z} and its corresponding image feature sampling point q i ={u, v} to construct more rich image features. The specific formula is as follows:

[0047]

[0048] Where, represents the LiDAR feature point p i as the query, each LiDAR feature point is projected to the image plane, and its corresponding sampling point q i is taken as the key K and the value V in . F enhance_c represents the initial enhanced feature, represents the element of the LiDAR feature at the index position (i, j), Atten(·) represents the attention mechanism, represents the element of the image feature at the index position (i, j), d k represents the dimension of the key vector.

[0049] In the step of "fusing the original RGB image features and the enhanced RGB image features through a residual network to obtain the RGB image features under the bird's eye view", the fine interaction unit includes two residual modules, which fuse the original RGB image features and the enhanced RGB image features through a residual network to obtain the RGB image features under the bird's eye view.

[0050] Specifically, the first residual module is responsible for preliminary processing of the enhanced image features, extracting features through convolution layers, batch normalization and ReLU activation functions, etc. The second residual module further processes the features to ensure the preservation of key information and the enhancement of feature representation. The specific formula is as follows:

[0051] F final_c =F sec (F first (F enhance_c )) (3)

[0052] Where F(x) represents the residual function, F first (x) represents the feature output after the first residual unit, and x is the input feature.

[0053] Through the above unit, the implicit spatial enhancement fusion coding module can effectively utilize the LiDAR data to provide accurate spatial information, and enhance the image features through implicit depth estimation, thereby improving the accuracy and robustness of three-dimensional target detection.

[0054] In the present scheme, the structure of the space-time interactive feature fusion network is as shown in Figure 4 The space-time interactive feature fusion network is described as follows:

[0055] In the step of "applying the deformable cross attention to the LiDAR point cloud features and the RGB image features under the bird's eye view in the spatial fusion unit to obtain the preliminary bird's eye view features", the spatial fusion unit includes a query generation and position encoding layer, a deformable cross attention layer and a residual layer. The query generation and position encoding layer generates query vectors corresponding to specific spatial positions in the ROI grid and encodes the positions to capture the relative spatial information of these positions. The deformable cross attention layer uses these query vectors to fuse the point cloud bird's eye view features and the image bird's eye view features. Finally, the residual layer performs residual connection to obtain the preliminary bird's eye view fusion features.

[0056] Specifically, the query vector is introduced in the spatial dimension, and the cross-attention mechanism is used to adaptively fuse the RGB image and LiDAR features, so as to maximize the complementary advantages of different modalities and capture spatial information from multiple angles. First, the spatial region of the region of interest (ROI) is divided into a grid array of HxW, and each grid cell corresponds to a specific position in the ROI space. A set of learnable parameters Q is introduced as the query vector, each query vector q i is mapped to a specific grid cell and is position-encoded based on its position information in the spatial coordinates, so that the model can understand the relative spatial information of each query vector, thereby more accurately locating and identifying objects during feature fusion. In addition, a spatial cross-attention mechanism based on deformable attention is introduced to minimize the dependence on computing resources. The specific formula is as follows:

[0057] Q i =CA(B c ,B l )=DefAtteni(B c ,B l ) (4)

[0058] where B l represents the LiDAR feature, and B c represents the final image feature. The encoded feature is divided into N regular grids, and each grid point corresponds to a query vector Q i . DefAtteni(·) represents the deformable attention mechanism. CA(·) represents the cross-attention mechanism.

[0059] In the step of "using a deformable self-attention mechanism in the temporal fusion unit to aggregate the preliminary bird's eye view feature to obtain the final fusion feature", the temporal fusion unit includes an initialization layer, a self-attention update layer, a feedforward network layer, and a fusion layer. The preliminary bird's eye view fusion feature is initialized using a self-attention mechanism in the initialization layer; then, in the self-attention update layer, the encoded features of the historical frames are fused into the query vector to update the query vector. The updated query vector is processed by the feedforward network layer to further enrich its feature representation, and finally the temporal information is fused by the fusion layer to obtain the final bird's eye view fusion feature.

[0060] Specifically, the preliminary bird's eye view fusion feature is initialized using a self-attention mechanism in the initialization layer based on the features of the current frame; then, in the self-attention update layer, the encoded features of the historical frames are fused into the query vector to update the query vector. The specific formula is as follows:

[0061]

[0062] wherein SA(·) represents a self-attention mechanism for initializing the query vector of the current frame. SA(·) represents a self-attention mechanism. DefAtteni(·) represents a deformable attention mechanism. FFN(·) represents a feed-forward network, represents the i-th query vector of the previous frame, F t represents the query vector of frame t, F t represents the encoded feature of the current frame.

[0063] Further, the updated query vector is calculated and processed by a time attention mechanism in the feed-forward network layer to further enrich its feature representation, and finally the temporal information is fused by the fusion layer to obtain the final bird's eye view fusion feature. The specific formula is as follows:

[0064]

[0065] wherein TSA(·) represents a time self-attention mechanism. FFN(·) represents a feed-forward network, DefAtteni(·) represents a deformable attention mechanism, Q t represents the query vector of frame t, F represents the i-th query vector of the current frame, F t-j represents the encoded feature of the previous frame, F represents the final temporal encoded feature.

[0066] Specifically, the space-time interactive feature fusion network establishes the connection between the two modal features by applying deformable attention mechanisms in different dimensions through the above-mentioned spatial fusion unit and time fusion unit, and aggregates semantic information from the multi-modal context, thereby obtaining an enhanced final feature representation. By introducing a learnable query vector and a spatial cross-attention mechanism in the spatial dimension, the preliminary BEV feature generation of the RGB image and the LiDAR feature is realized. Subsequently, in the time dimension, the self-attention mechanism and the feed-forward network are used to integrate and multi-layer time fusion encode the historical frame features, thereby obtaining the final fusion feature.

[0067] Specifically, the final fusion feature is input into a classification head, which performs bounding box regression, attribute and classification prediction to obtain a 3D target detection result, and a loss function is constructed based on the detection result. When the loss function meets the set condition, the current parameters of the 3D target detection architecture are saved.

[0068] Specifically, the data in the nuScenes dataset is used as the training sample and the validation set, and the robustness and effectiveness of the model are proved through the validation of the 3D target detection model.

[0069] Embodiment Two

[0070] The embodiment relates to a 3D target detection device based on inter-modal implicit space enhancement and space-time interaction coding, comprising a memory and one or more processors, the memory stores executable codes, and the one or more processors execute the executable codes to implement the 3D target detection method based on inter-modal implicit space enhancement and space-time interaction coding of the embodiment 1.

[0071] Embodiment three

[0072] The embodiment relates to a computer readable storage medium, which stores a program, and the program is executed by a processor to implement the 3D target detection method based on inter-modal implicit space enhancement and space-time interaction coding of the embodiment two.

[0073] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A 3D object detection method based on inter-modal implicit spatial enhancement and spatio-temporal interaction encoding, characterized in that, The method comprises the following steps: S1: Obtain one-to-one corresponding multi-view RGB image data and LiDAR point cloud data as training samples, and construct a 3D target detection architecture, wherein the 3D target detection architecture comprises a feature extraction network, an implicit space enhancement fusion coding network, a space-time interactive feature fusion network and a classification head; S2: Input a set of corresponding multi-view RGB image data and LiDAR point cloud data in the training samples into the feature extraction network for feature extraction to obtain RGB image features and LiDAR point cloud features; S3: The implicit space enhancement fusion coding network comprises a preliminary interaction unit and a fine interaction unit, in the preliminary interaction unit, the LiDAR point cloud data is projected into the RGB image coordinate system by using the intrinsic matrix and the extrinsic matrix of the camera, and the spatial depth information of the RGB image features is dynamically enhanced by using a variable cross-attention mechanism, in the fine interaction unit, the original RGB image features and the enhanced RGB image features are fused by using a residual network to obtain the RGB image features in the bird's eye view; S4: Input the LiDAR point cloud features and the RGB image features in the bird's eye view into the space-time interactive feature fusion network, wherein the space-time interactive feature fusion network comprises a space fusion unit and a time fusion unit, in the space fusion unit, the LiDAR point cloud features and the RGB image features in the bird's eye view are applied to a deformable cross-attention to obtain preliminary bird's eye view features, in the time fusion unit, the preliminary bird's eye view features are aggregated by using a deformable self-attention mechanism to obtain final fusion features; The space fusion unit comprises a query generation and position encoding layer, a deformable cross-attention layer and a residual layer; the query generation and position encoding layer generates query vectors corresponding to specific spatial positions in the ROI grid and encodes the positions to capture the relative spatial information of the positions; the deformable cross-attention layer fuses the point cloud bird's eye view features and the image bird's eye view features by using the query vectors; finally, the residual layer performs residual connection to obtain preliminary bird's eye view fusion features; The time fusion unit comprises an initialization layer, a self-attention update layer, a feedforward network layer and a fusion layer; the preliminary bird's eye view fusion features initialize query vectors by using a self-attention mechanism in the initialization layer; then, the query vectors are updated by using a deformable attention mechanism to fuse the encoding features of the historical frames into the query vectors in the self-attention update layer; the updated query vectors are processed by the feedforward network layer to further enrich the feature representation, and finally, the time sequence information is fused by the fusion layer to obtain the final bird's eye view fusion features; S5: Input the final fusion features into the classification head to obtain 3D target detection results, construct a loss function based on the detection results, save the current parameters of the 3D target detection architecture when the loss function meets the set condition, and obtain a 3D target detection model; S6: Input a set of corresponding multi-view RGB image data and LiDAR point cloud data into the trained 3D target detection model to obtain target detection results.

2. The 3D object detection method based on inter-modal implicit spatial enhancement and spatio-temporal interaction encoding according to claim 1, characterized in that, The preliminary interaction unit in step S3 includes a projection operation layer and a spatial coding layer; in the projection operation layer, the LiDAR point cloud data is projected onto the RGB image features by using the intrinsic matrix and the extrinsic matrix of the camera to establish a coordinate system correspondence; in the spatial coding layer, the variable cross-attention mechanism is combined with the accurate spatial information of the LiDAR data to perform operation to obtain enhanced image features.

3. The 3D object detection method based on inter-modal implicit spatial enhancement and spatio-temporal interaction encoding according to claim 1, characterized in that, The fine interaction unit in step S3 includes two residual modules, and the original RGB image features are fused with the enhanced RGB image features by using the residual network to obtain the RGB image features in the bird's eye view.

4. The 3D object detection device based on inter-modal implicit spatial enhancement and space-time interaction coding has the characteristics that The memory and one or more processors are included, executable codes are stored in the memory, and the one or more processors execute the executable codes to implement the 3D object detection method based on inter-modal implicit spatial enhancement and space-time interaction coding in any one of claims 1-3.

5. A computer readable storage medium, characterized in that, A program is stored thereon, and the program is executed by a processor to implement the 3D object detection method based on inter-modal implicit spatial enhancement and space-time interaction coding in any one of claims 1-3.

Images