LSTM input fusion architecture design method, electronic devices and storage media

Abstract

This application provides an LSTM input fusion structure design method, electronic device, and storage medium, belonging to the field of LSTM state computation technology. The method includes: acquiring the feature frame of the current task; expanding the feature frame into a sliding window sequence according to the time dimension based on a preset channel mapping method; using the sliding window sequence as the input sequence at the current time step; and constructing the target LSTM input vector of the current task based on the data concatenation rules between the input sequence at the current time step and the hidden state at the previous time step. This application's scheme designs an LSTM input fusion structure suitable for CNN-LSTM joint models.

Description

Technical Field

[0001] This application relates to the field of LSTM state computation technology, specifically to an LSTM input fusion structure design method, electronic device, and computer-readable storage medium. Background Technology

[0002] With the widespread deployment of GNSS (Global Navigation Satellite System) in vehicles, drones, and smart terminals, more and more navigation systems are beginning to incorporate lightweight artificial intelligence technologies. Among them, CNN (Convolutional Neural Network) and LSTM (Long Short-Term Memory) are two typical deep learning structures, respectively responsible for spatial feature extraction and time series modeling.

[0003] Currently, existing technologies attempt to combine CNN and LSTM models to construct a fusion structure of "space + time". However, in actual chip deployment, CNN output is usually a two-dimensional feature map, while the input required by the LSTM inference model is a time-series vector stream, requiring data to be input in an ordered manner according to time steps. This structural mismatch becomes a major obstacle to system integration.

[0004] Therefore, how to organize the two-dimensional feature maps or GNSS sliding window feature maps output by CNN into a time-series format vector stream that can be directly read by the LSTM inference model in a GNSS SoC chip system, so as to reduce the latency and power consumption caused by cross-module data transfer, reduce the complexity of the system data path, and improve the inference efficiency of the SoC chip system, has become an unsolved problem. Summary of the Invention

[0005] The purpose of this application is to provide an LSTM input fusion structure design method, electronic device, and computer-readable storage medium, which can solve the problem of mismatch between the structural forms of CNN and LSTM in the prior art.

[0006] In a first aspect, embodiments of this application provide a method for designing an LSTM input fusion structure, the method comprising:

[0007] Obtain the feature frames of the current task;

[0008] Based on a preset channel mapping method, the feature frames are expanded into a sliding window sequence according to the time dimension; the sliding window sequence serves as the input sequence for the current time step.

[0009] Based on the data concatenation rules between the current input sequence and the hidden state of the previous time step, construct the target LSTM input vector for the current task.

[0010] In one possible implementation of the first aspect, the preset channel mapping method includes: a first mapping method, wherein the feature frames include: CNN-derived frames and navigation system native sliding window frames; based on the preset channel mapping method, the feature frames are expanded into a sliding window sequence according to the time dimension, including:

[0011] Based on the sliding window frame order or channel interleaving method, each CNN output channel is concatenated with each time step to obtain the sliding window sequence;

[0012] Each CNN output channel is mapped to a sliding window sequence.

[0013] In one possible implementation of the first aspect, the preset channel mapping method includes: a second mapping method; based on the preset channel mapping method, expanding the feature frames into a sliding window sequence according to the time dimension includes:

[0014] Load the valid CNN output channels based on the channel mask configured for the current task;

[0015] Depending on the sliding window frame order or channel interleaving method, the effective CNN output channels are concatenated with each time step to obtain the sliding window sequence;

[0016] In this process, the effective CNN output channels are mapped to a sliding window sequence.

[0017] In one possible implementation of the first aspect, the method further includes:

[0018] The data bit width of the effective CNN output channel is compressed according to the preset compression precision mode.

[0019] In one possible implementation of the first aspect, the preset compression precision mode includes: INT8 mode and INT4 mode; according to the preset compression precision mode, the data bit width of the effective CNN output channel is compressed, including:

[0020] Based on the INT8 mode, the data bit width of the effective CNN output channel is compressed to INT8;

[0021] Alternatively, based on the INT4 mode, the data bit width of the effective CNN output channels can be compressed to INT4.

[0022] In one possible implementation of the first aspect, the target LSTM input vector for the current task is constructed based on the data concatenation rules between the current input sequence and the hidden state of the previous time step, including:

[0023] The current input sequence is concatenated or arranged side-by-side with the hidden state from the previous time step, and the concatenation or arrangement result is used as the target LSTM input vector for the current task.

[0024] In one possible implementation of the first aspect, the target LSTM input vector for the current task is constructed based on the data concatenation rules between the current input sequence and the hidden state of the previous time step, including:

[0025] Use the current input sequence as the target LSTM input vector for the current task;

[0026] In this case, the input sequence at the current time step is not concatenated or arranged in parallel with the hidden state at the previous time step.

[0027] In one possible implementation of the first aspect, the method further includes:

[0028] Configure the sliding window size, channel mask, input shape transformation method, data bit width, and activation flag based on the current task number.

[0029] Secondly, embodiments of this application provide an LSTM input fusion structure design apparatus, the apparatus comprising:

[0030] The acquisition unit is used to acquire the feature frames of the current task;

[0031] The expansion unit is used to expand the feature frame into a sliding window sequence according to the time dimension based on a preset channel mapping method; the sliding window sequence serves as the input sequence at the current time.

[0032] The construction unit is used to construct the target LSTM input vector for the current task based on the data concatenation rules between the current input sequence and the hidden state of the previous time step.

[0033] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the LSTM input fusion structure design method of any of the first aspects described above.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the LSTM input fusion structure design method of any one of the first aspects described above.

[0035] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to execute the LSTM input fusion structure design method of any one of the first aspects described above.

[0036] The proposed solution first obtains the feature frames of the current task, then expands the feature frames into a sliding window sequence in the time dimension based on a preset channel mapping method, and finally constructs the target LSTM input vector of the current task based on the data concatenation rules of the input sequence (sliding window sequence) at the current time and the hidden state at the previous time.

[0037] This application's solution organizes the two-dimensional feature map output by CNN or GNSS sliding window feature map into a time-series format vector stream that can be directly read by the LSTM inference model through sliding window expansion and data stitching in the GNSS SoC chip system. No master control rearrangement or DMA data transfer is required between CNN and LSTM, which can reduce the latency and power consumption caused by cross-module data transfer and reduce the complexity of system data path. It can also directly complete the rearrangement, stitching and format adaptation of feature map to sequence input within the chip, achieving seamless input form connection, path reuse and energy efficiency optimization, thereby improving the inference efficiency of SoC chip system under GNSS perception mission.

[0038] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram illustrating the steps of the LSTM input fusion structure design method provided in the embodiments of this application;

[0041] Figure 2 This is a schematic diagram of the sliding window feature map construction and row block rearrangement provided in the embodiments of this application;

[0042] Figure 3 This is a schematic diagram showing an overview of the CNN output feature map to LSTM input fusion structure provided in the embodiments of this application;

[0043] Figure 4 This is a schematic diagram of the input fusion path data flow provided in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the configuration and control interface structure provided in the embodiments of this application;

[0045] Figure 6 This is a schematic diagram of the LSTM input fusion structure design device provided in the embodiments of this application;

[0046] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0047] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0048] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or photovoltaic modules, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, photovoltaic modules and / or combinations thereof.

[0049] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0050] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0051] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [the described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [the described condition or event] is detected," or "in response to detection of [the described condition or event]."

[0052] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in some other embodiments," "in other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0054] With the widespread deployment of GNSS in vehicles, drones, and smart terminals, more and more navigation systems are incorporating lightweight artificial intelligence technologies to enhance the intelligence and environmental adaptability of navigation signal processing. Among them, CNN and LSTM, as two typical deep learning structures, are responsible for spatial feature extraction and time series modeling, respectively, and are being used in practical applications in GNSS intelligent sensing scenarios such as interference identification, multipath modeling, and signal loss detection.

[0055] Currently, many embedded navigation chip solutions attempt to combine CNN and LSTM models to construct a "space + time" fusion structure. However, in actual chip deployment, CNN output is typically a two-dimensional feature map, such as a channel spectrogram or a pseudorange sliding window residual map, with a typical output format of H×W×C (height × width × number of output channels). LSTM modules, on the other hand, require a time-series format vector stream (T×D, time scale × number of sparse channels), demanding ordered data input step-by-step. This structural mismatch becomes a major obstacle to system integration.

[0056] Currently, mainstream technologies typically employ the following methods for data exchange between CNN and LSTM:

[0057] (1) Main CPU reassembles input: After the CNN inference is completed, the feature map is moved to the main memory, and the CPU rearranges, transforms and sends it to the LSTM module in sequence;

[0058] (2) External DMA (Direct Memory Access) transfer buffer: A DMA module is inserted between the CNN output and the LSTM input to perform sliding window construction and row and column rearrangement operations;

[0059] (3) Software decoupling and asynchronous execution: CNN and LSTM are decoupled into two modules, and the system software controls the task boundaries, synchronizes the results and recombines the inputs.

[0060] However, the above methods have the following problems on platforms such as GNSS SoC (System on Chip), which are highly sensitive to power consumption, area, and response speed:

[0061] (1) Frequent master control intervention and high system energy consumption: CPU participates in data preprocessing and format conversion, which seriously occupies control resources;

[0062] (2) Severe buffer redundancy, increasing area and latency: intermediate data needs to be cached and copied more than twice, resulting in broken inference paths;

[0063] (3) Strong interface heterogeneity and complex model deployment: The connection between CNN and LSTM requires writing a lot of Glue Code, which affects the automatic deployment process;

[0064] (4) It cannot run smoothly and it is difficult to achieve end-to-end optimization: Interrupted data exchange hinders the efficient deployment of CNN-LSTM cascade structures.

[0065] Currently, the following technical problems exist in the deployment of the CNN-LSTM joint model in existing GNSS chips:

[0066] (1) Data structure incompatibility: CNN output is usually a two-dimensional feature map, while the input required by the LSTM module is a vector stream in time series format, which is structurally incompatible;

[0067] (2) Input interface separation: The input interface is separated from the CNN output and cannot be directly rearranged into an input format that conforms to the LSTM inference structure;

[0068] (3) High energy consumption and large inference delay: The CPU participates in data preprocessing and format conversion, which seriously occupies control resources.

[0069] Therefore, how to design an LSTM input fusion structure suitable for CNN-LSTM joint models has become an unsolved problem.

[0070] To address the aforementioned shortcomings, this application provides an LSTM input fusion structure design method. First, the feature frames of the current task are obtained. Then, based on a preset channel mapping method, the feature frames are expanded into a sliding window sequence along the time dimension. Finally, the target LSTM input vector of the current task is constructed based on the data concatenation rules of the current input sequence (sliding window sequence) and the hidden state of the previous time step.

[0071] This application proposes an LSTM input fusion structure, namely the LSTM input vector, suitable for CNN-LSTM joint models, through sliding window expansion and data concatenation rules. This structure is highly user-friendly and practical.

[0072] The specific process implemented in this application is described below through specific embodiments.

[0073] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the steps of the LSTM input fusion structure design method provided in the embodiments of this application. For example... Figure 1 As shown, the method includes the following steps:

[0074] S101, Obtain the feature frame of the current task.

[0075] In one embodiment, the current task can be interference prediction, multipath classification, signal drift modeling, etc. The feature frames include: CNN-derived frames and navigation system native sliding window frames. The CNN-derived frames are two-dimensional feature maps output by the CNN module, and the navigation system native sliding window frames are GNSS sliding window feature maps.

[0076] It should be noted that the two-dimensional feature map output by the CNN module is a learned or derived feature extracted from the input (e.g., correlation power spectrum, residual map, etc.) by operators such as convolution or pooling. Its shape is often denoted as H×W×C or flattened into "number of rows (W) × number of output channels (C)". GNSS sliding window feature maps usually refer to time-frequency or time-channel observation arrays directly constructed by receiver front-end / correlator / frequency domain processing (e.g., multi-correlation early / mid / late taps, stacking of spectrum / residual over time), which are more "raw or lightly processed features".

[0077] In one embodiment, the two-dimensional feature map output by the CNN module and the GNSS sliding window feature map are not the same concept, but they can be treated uniformly. For the "input fusion structure," both can be regarded as "two-dimensional frames arriving over time." They are collectively referred to as "feature frames," which can originate from either "CNN-derived frames" or "GNSS native sliding window frames."

[0078] It should be noted that this application is applicable to GNSS systems, and the data structure of GNSS systems has been optimized.

[0079] S102, based on a preset channel mapping method, the feature frame is expanded into a sliding window sequence according to the time dimension; the sliding window sequence serves as the input sequence at the current moment.

[0080] In one embodiment, the two-dimensional feature map output by the CNN module or the two-dimensional data in the GNSS sliding window feature map is expanded into a sliding window sequence along the time dimension (e.g., combined into a T×D structure by W×C columns) to form an input buffer structure that can be continuously read by the LSTM.

[0081] In one embodiment, there can be multiple sliding window sequences, with different time scales (e.g., T1=2, T2=4, T3=8) forming multi-scale sliding window sequences; different time lengths / overlap rates (stride, s) form different time resolutions; different channel subsets (MASK) are geared towards different tasks, such as interference detection, multipath classification, signal loss detection, etc.

[0082] In one embodiment, the current input sequence x(t) is defined and assigned as follows: for a single configuration (a certain task ID, a specific T / MASK / stride), x(t) is the sliding window expansion result of that configuration at the current time t (in the form of a flattened T×D vector); when there are multiple sliding window sequences, there are three main supply methods (all supported in this application, selected by the configurator):

[0083] (1) Parallel concat: Concatenates multiple sliding window sequences into a larger sequence in the channel dimension. , Input the same LSTM at once (D increases, timing is consistent); (2) Multiple heads / multiple instances: Each sliding window sequence is connected to an LSTM head, and then fused at the decision layer (suitable for tasks with large differences); (3) Time division multiplexing (Task_ID rotation): Multiple tasks quickly switch context, and the input fusion module outputs x(t) corresponding to different sliding window sequences in different time slots. This method can improve throughput and hardware reuse rate.

[0084] In one embodiment, multiple parallel or time-division sliding window sequences are configured based on different sliding window lengths T, step sizes s, and channel masks. In parallel mode, the input fusion structure concatenates the multiple sliding window sequences along the channel dimension to form a single LSTM input vector x(t); in multi-instance or time-division multiplexing modes, the corresponding x is output respectively. i (t) output to different LSTM instances or to the same LSTM instance in adjacent time slots.

[0085] According to one embodiment of this application, the preset channel mapping method can be a first mapping method, i.e., full-channel mapping, which can map all CNN output channels to a sliding window sequence. Based on the preset channel mapping method, the feature frames are expanded into a sliding window sequence according to the time dimension, including:

[0086] Depending on the sliding window frame order or channel interleaving method, each CNN output channel is concatenated with each time step to obtain a sliding window sequence; where each CNN output channel is mapped to the sliding window sequence.

[0087] Please see Figure 2 , Figure 2 This is a schematic diagram of the sliding window feature map construction and row / block rearrangement provided in an embodiment of this application. For example... Figure 2 As shown, the CNN module outputs the channel spectrogram or pseudorange sliding window residual map (not shown in the figure) as a two-dimensional feature map, with a shape assumed to be W×C, where W represents the sliding window width, C represents the number of output channels, and the C output channels are ch1~chC, forming the feature map row at the current time step. At each time step, the CNN module outputs a row of feature vectors, showing the multi-channel features of the current frame.

[0088] In one embodiment, such as Figure 2 As shown, the sliding window step size is 4 frames, and the sliding window expands to t-3, t-2, t-1, t. According to the sliding window row frame order (the order of t-3 to t), each CNN output channel (ch1 to chC) is concatenated with each time step (t-3 to t) to generate a sliding window sequence. This sliding window sequence serves as the input sequence for the current time step. , is a one-dimensional vector [1×C]. All CNN output channels (ch1~chC) are mapped to a sliding window sequence.

[0089] In one embodiment, the channel data sequence within the sliding window is flattened into a one-dimensional vector based on time step t~(t-3); multiple frames of historical channel data are concatenated into the expected input of the LSTM (enhanced temporal correlation); and adjustable configurations such as interval sampling, channel filtering, and bit width compression are supported.

[0090] In one embodiment, a sliding window mechanism is used to retain the CNN output of the past N frames. After each update, the following steps are performed: the sliding window buffer stores the most recent N rows of output, and the oldest row is discarded when a new row arrives; the sliding window row block rearrangement module concatenates the contents stored in the sliding window buffer into a C structure (feature map row) in row-frame order; if there is a channel mask, only some channels are extracted and concatenated to generate a 1xD vector.

[0091] In another embodiment, when generating the sliding window sequence using a channel interleaving method, the splicing is not performed according to the order of the sliding window frames. Instead, the order of splicing the CNN output channels and time steps is adjusted before splicing. For example, instead of splicing ch1@t-3, ch1@t-2 can be spliced ​​after ch1@t-3, ch5@t-2 or other channel splices can be spliced. Alternatively, the order of the CNN output channels or time steps can be adjusted before splicing, for example, the order of the time steps can be adjusted to t-2, t, t-1, t-3.

[0092] According to another embodiment of this application, the preset channel mapping method can also be a second mapping method, namely channel sparse mapping, which sparsely extracts (e.g., extracts at intervals of several channels) a portion of the CNN output channels and maps them to the sliding window sequence. That is, among all the CNN output channels, only a portion of the effective channels are selected according to the configuration, mapped and concatenated to the input sequence x(t).

[0093] According to another embodiment of this application, based on a preset channel mapping method, the feature frames are expanded into a sliding window sequence according to the time dimension, including:

[0094] Based on the channel mask configured for the current task, load the effective CNN output channels; according to the sliding window frame order or channel interleaving method, concatenate the effective CNN output channels with each time step to obtain the sliding window sequence; wherein, the effective CNN output channels are mapped to the sliding window sequence.

[0095] In some embodiments, the total number of CNN output channels is 16. If the channel mask is 0x00FF, the effective CNN output channels are the last 8 (ch9~ch16); if the channel mask is 0x0F0F, the effective CNN output channels are ch5~ch8 and ch13~ch16.

[0096] In one embodiment, based on the sliding window frame order or channel interleaving method, a portion of the channels (effective CNN output channels) are extracted and concatenated with each time step to generate a sliding window sequence. The specific concatenation method is essentially the same as the concatenation method corresponding to the full channel mapping method in the above embodiment, and will not be described again here.

[0097] In one embodiment, the two-dimensional feature map output by the CNN module has W=4 and C=16. Each input takes 4 frames of channel data, which are then concatenated to form a one-dimensional vector x(t) with T=4 and D=8 (sparse channel number), and the structure is [1×D].

[0098] In one embodiment, valid CNN output channels are filtered according to a configured channel mask, and only the enabled channel data is loaded for each inference, reducing the number of channels D and decreasing data volume and energy consumption.

[0099] In this embodiment, a channel sparse mapping method is adopted, which selectively loads effective feature channels according to the model configuration parameters (channel mask), thereby reducing redundant data handling and buffering overhead.

[0100] According to one embodiment of this application, the method further includes: compressing the data bit width of the effective CNN output channel according to a preset compression precision mode.

[0101] According to one embodiment of this application, the preset compression precision modes include: INT8 (data bit width is an 8-bit integer) mode and INT4 (data bit width is a 4-bit integer) mode. Based on the preset compression precision mode, the data bit width of the effective CNN output channel is compressed, including:

[0102] Based on INT8 mode, the data bit width of the effective CNN output channel is compressed to INT8; or, based on INT4 mode, the data bit width of the effective CNN output channel is compressed to INT4.

[0103] In one embodiment, bit width compression control is performed: the data of the effective CNN output channel can be converted from 32-bit float to INT8 or INT4 according to actual needs; the precision controller calls the CRC (Cyclic Redundancy Check) lookup table or right shift logic (shifting the binary bits of the value to the right by a specified number of bits, filling the left side with 0) to compress the bit width based on the model quantization (precision conversion) result; and the splicing result is aligned and padded to adapt to the LSTM input format.

[0104] In this embodiment, by using a bit-width compression mechanism, INT8 or INT4 low-bit compression is performed according to the model configuration parameters (data bit width), which can reduce redundant data handling and buffering overhead.

[0105] S103: Based on the data concatenation rules between the current input sequence and the hidden state of the previous time step, construct the target LSTM input vector for the current task.

[0106] In one embodiment, the data concatenation rule is as follows: It selects whether the current input sequence x(t) is concatenated or arranged side-by-side with the hidden state h(t-1) from the previous time step, and the concatenation position. Here, concatenation refers to joining the two vectors x(t) and h(t-1) end-to-end along their feature dimensions. , After splicing ( and The order of these elements is determined by the configuration. The splicing position refers to... and The order of precedence / the range of characteristic subscripts .

[0107] For example, if x_first=1 is configured, then , If x_first=0 is configured, the order is reversed.

[0108] In one embodiment, for a side-by-side arrangement, if the two vectors x(t) and h(t-1) are formed into a two-dimensional matrix... This is a representation of column stacking, but the standard input to an LSTM is usually a one-dimensional vector, which is ultimately flattened into... In engineering implementation, one-dimensional splicing is performed directly, eliminating one reshape (matrix transformation function) / transpose (transpose).

[0109] For example, , If the splicing order is ,but If the splicing order is ,but .

[0110] According to one embodiment of this application, the target LSTM input vector for the current task is constructed based on the data concatenation rules between the current input sequence and the hidden state of the previous time step, including:

[0111] The current input sequence is concatenated or arranged side-by-side with the hidden state from the previous time step, and the concatenation or arrangement result is used as the target LSTM input vector for the current task.

[0112] In one embodiment, the feature concatenation and timing shaper automatically concatenates or arranges the current input sequence x(t) with the hidden state h(t-1) from the previous time step according to the model configuration to construct a standard LSTM input vector. The model configuration mainly refers to the parameter settings written to the register by the main controller or model loader during the chip initialization or task loading phase, including: (1) whether to concatenate: whether to use only the current frame x(t) or to concatenate the hidden state h(t-1) from the previous time step for the LSTM input vector; (2) concatenation position and shape: concatenation order and structural description of the output vector (Input_Shape) to ensure matching with the input interface of the target LSTM core.

[0113] In one embodiment, the final LSTM input vector consists of two parts: (1) the current frame CNN sliding window feature vector: x(t) after sparse channel selection, bit width compression and flattening; (2) the previous hidden state h(t-1), which is the storage result from the state bank.

[0114] In one embodiment, the splicing controller is activated to splice x(t) and h(t-1), resulting in the form x(t)‖h(t-1), which is then input into the LSTM computation core.

[0115] According to another embodiment of this application, the target LSTM input vector for the current task is constructed based on the data concatenation rules between the current input sequence and the hidden state of the previous time step, including:

[0116] The current input sequence is used as the target LSTM input vector for the current task; the current input sequence is not concatenated or arranged side by side with the hidden state of the previous time step.

[0117] In one embodiment, if the model structure does not require splicing, the controller can be configured to "single-input mode" with only x(t) as input.

[0118] According to one embodiment of this application, the method further includes: configuring the sliding window size, channel mask, input shape transformation method, data bit width, and activation flag according to the current task number.

[0119] In one embodiment, the multi-task input configuration controller supports configuring the sliding window size, channel mask, input shape transformation method, data bit width, and activation flag by task number, accommodating the concurrent execution requirements of multiple GNSS tasks. The input shape transformation method is an expansion into a sliding window sequence, and the activation flag is a bit field used to enable / select a specific processing branch (including but not limited to the precision path), which can both switch splicing branches on / off and select the precision level.

[0120] In one embodiment, the activation flag serves as an enable / mode bit for the fusion controller, used to turn optional sub-precision paths on or off. Whether it's used to "activate a precision path" depends on the register definition. Typically, a set of bits controls different functions, for example, as follows:

[0121] EN_MASK: Enable channel masking (channel sparse mapping); EN_COMP: Enable bit width compression (INT8 / INT4); EN_CAT_H: Enable concatenation with h(t-1); EN_CAT_XPREV: Enable concatenation with x(t); EN_DIRECT: x(t) pass through (no concatenation); PREC_SEL[1:0]: Activate the flag / selection bits of a certain precision path for precision path selection, such as 00=FP16, 01=INT8, 10=INT4.

[0122] It should be noted that all fusion processing at the input path pass-through interface is completed directly between the CNN module and the LSTM inference path, without the need for interrupt / mediator controllers, forming a cascaded computation chain. The entire fusion process is completed within one system clock cycle, without the need for master control intervention.

[0123] Please see Figure 3 , Figure 3 This is a schematic diagram showing the overall structure of the CNN output feature map to LSTM input fusion architecture provided in this application embodiment. The diagram illustrates the input fusion path between the CNN and LSTM modules and their integrated structure within the SoC. Figure 3 As shown, the CNN module is upstream, the LSTM state calculation unit is downstream, and the LSTM input fusion structure designed in this application is between the two.

[0124] In one embodiment, the CNN module outputs a two-dimensional feature map with a shape of H×W×C. The LSTM state computation unit is used to perform LSTM state computation.

[0125] In one embodiment, the LSTM input fusion structure includes: a sliding window row block rearrangement module, a channel order mapping module, and an LSTM input vector generation unit. The sliding window row block rearrangement module is used to generate a sliding window sequence; the channel order mapping module is used to perform channel sparse mapping according to the channel mask and the sliding window row frame order, wherein the order mapping is to unfold and concatenate the channel data sequentially according to the time order of the sliding window row frames, so that the generated input sequence can be continuously read by the LSTM step by step.

[0126] In one embodiment, the LSTM input vector generation unit is used to generate the input sequence x(t) (T×D vector) at the current time. x(t) can be directly input into the LSTM state calculation unit, or concatenated with h(t-1) (x(t)‖h(t-1)) and then input into the LSTM state calculation unit.

[0127] In one embodiment, the timing splicer is used to convert feature maps into timing sequences, the bit width compression module is used to perform bit width compression, and the Flatten controller is used to perform flattening control, flattening the channel data sequence within the sliding window into a one-dimensional vector.

[0128] According to one embodiment of this application, initialization and structure configuration are performed: After system startup, the main controller or model loader initializes and configures the input fusion structure through the APB (Advanced Peripheral Bus) or CSR (Certificate Signing Request) interface. The configuration items are as follows:

[0129] Window Length: For example, set to 4 frames; Channel Mask: Used to filter valid CNN output channels; Precision Mode: For example, INT8 or INT4; Flatten Mode: Can be selected to straighten by time step or interleave by channel; Input Fusion Mode: Selects whether x(t) is concatenated with h(t-1), the concatenation position, etc.; Input Shape Specification: The data structure description before outputting the LSTM. These configurations are written to the register page and locked, and the input fusion structure then enters the "fusion ready" state.

[0130] Please see Figure 4 , Figure 4This is a schematic diagram of the input fusion path data flow provided in an embodiment of this application. For example... Figure 4 As shown, the process includes the following steps:

[0131] (1) CNN feature map output;

[0132] (2) Sliding window construction and buffer storage of the most recent N rows of CNN output;

[0133] (3) The channel mapping controller performs channel mask loading and channel rearrangement index mapping;

[0134] (4) The bit width compression module performs bit width compression, which can compress the bit width of the channel data to INT8 / INT4;

[0135] (5) The timing splicer and flattener perform timing splicing and flattening control, and finally generate the LSTM input vector x(t).

[0136] It should be noted that the specific implementation principle of this embodiment has been described in the above embodiments, and will not be repeated here.

[0137] Please see Figure 5 , Figure 5 This is a schematic diagram of the configuration and control interface structure provided in an embodiment of this application. For example... Figure 5 As shown, the configuration of the sliding window length, channel mask, data bit width, input shape integration mode, and control interface is illustrated.

[0138] In one embodiment, the main control CPU / configuration engine is connected to the input fusion structure configuration controller via the APB / CSR interface. The input fusion structure configuration controller is used for: (1) configuring the sliding window length and controlling the rearrangement logic; (2) loading the channel mask and controlling the channel mapping module; (3) setting the data bit width and controlling the compression path; and (4) configuring the input shape integration mode and controlling the timing splicer.

[0139] It should be noted that the control register is configured by the master controller and supports adaptive multi-task structure; it can dynamically switch the sliding window length, channel combination, and splicing strategy, and supports rapid adjustment at runtime; all configuration results take effect automatically in the input fusion structure without the need for master controller to participate in relocation or synchronization.

[0140] According to one embodiment of this application, a multi-task adaptation example is as follows:

[0141] If the SoC runs multiple CNN-LSTM tasks simultaneously (such as interference prediction, multipath classification, signal drift modeling, etc.), the system can allocate multiple sets of fusion structure configuration items, as shown in Table 1. Table 1 is as follows:

[0142] Table 1

[0143]

[0144] Here, Task_ID is the task number. The system loads the corresponding configuration according to the task number through a context switching mechanism, enabling the coexistence of multiple task input paths.

[0145] According to one embodiment of this application, the actual measured power consumption and resource efficiency are described as follows:

[0146] Deployment scenario: The CNN output consists of 4 frames × 8 channels, and the LSTM input needs to be concatenated with the hidden state from the previous time step, totaling 64 bytes of input. Compared with traditional solutions (requiring caching + CPU reorganization), the resource and power consumption performance of the structure in this application is shown in Table 2. Table 2 is as follows:

[0147] Table 2

[0148]

[0149] The LSTM input fusion structure design method provided in this application first obtains the feature frame of the current task, then expands the feature frame into a sliding window sequence in the time dimension based on a preset channel mapping method, and finally constructs the target LSTM input vector of the current task based on the data concatenation rules of the input sequence (sliding window sequence) at the current time and the hidden state at the previous time.

[0150] This application proposes an LSTM input fusion structure, namely the LSTM input vector, suitable for CNN-LSTM joint models, through sliding window expansion and data concatenation rules. This structure is highly user-friendly and practical.

[0151] This application provides a unified input fusion structure for CNN-LSTM collaborative processing paths in GNSS chips. It is applicable to GNSS chip systems with CNN-LSTM joint inference paths and can directly complete the rearrangement, splicing, and format adaptation of feature maps to sequence inputs within the chip. This achieves seamless input form connection, path reuse, and energy efficiency optimization, thereby improving the inference efficiency and task response capability of the entire SoC system.

[0152] Compared to existing CNN-LSTM input adaptation methods, the proposed solution achieves the following key technological breakthroughs and enhanced engineering value on the GNSS SoC platform:

[0153] (1) The input structure is fully integrated, improving the module coupling: There is no need for master control rearrangement or DMA transfer between CNN and LSTM. Intermediate data can be directly read by LSTM through the sliding window mechanism, which greatly reduces the complexity of the system data path.

[0154] (2) Data reuse and low power optimization synergy: The input channel is loaded in a sparse configuration. Combined with a low bit width compression scheme, the amount of input data can be reduced while ensuring accuracy, thereby reducing the number of bus accesses and power consumption.

[0155] (3) Automatic input splicing improves pipeline efficiency: It supports the one-time generation of the input structure required by LSTM (such as splicing the current frame + historical state), which can form a stable time-series data stream input when combined with clock triggering;

[0156] (4) Adapt to multiple input types and enhance task versatility: Support the unified fusion of CNN output feature maps, multi-channel GNSS sliding window signals, and even low-frequency IMU (Inertial Navigation System) information inputs to build a highly scalable input structure generation path;

[0157] (5) Optimize the deployment process at the structural level: TFLite (TensorFlow Lite, a deep learning inference framework) can predefine the fusion structure configuration during the model conversion stage, which directly corresponds to the hardware input path, simplifying the model deployment process and improving the feasibility of one-click deployment.

[0158] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0159] Corresponding to the method in the above embodiments, Figure 6 This is a schematic diagram of the LSTM input fusion structure design device provided in the embodiments of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0160] Please see Figure 6 The device includes:

[0161] Acquisition unit 601 is used to acquire the feature frame of the current task;

[0162] The expansion unit 602 is used to expand the feature frame into a sliding window sequence according to the time dimension based on a preset channel mapping method; the sliding window sequence serves as the input sequence at the current time.

[0163] Construction unit 603 is used to construct the target LSTM input vector for the current task based on the data concatenation rules of the current input sequence and the hidden state of the previous time step.

[0164] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0166] Figure 7 This is a schematic diagram of the structure of the electronic device 7 provided in an embodiment of this application. Figure 7 As shown, the electronic device 7 of this embodiment includes: at least one processor 701 ( Figure 7 Only one is shown in the diagram), memory 703, and computer program 702 stored in memory 703 and executable on at least one processor 701, wherein processor 701 executes computer program 702 to implement the steps in the above method embodiments.

[0167] Electronic device 7 can be a desktop computer, laptop, handheld computer, or mobile phone, etc. This electronic device 7 may include, but is not limited to, a processor 701 and a memory 703. Those skilled in the art will understand that... Figure 7 This is merely an example of electronic device 7 and does not constitute a limitation on electronic device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0168] The processor 701 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware photovoltaic modules, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0169] In some embodiments, memory 703 may be an internal storage unit of electronic device 7, such as a hard disk or memory of electronic device 7. In other embodiments, memory 703 may be an external storage device of electronic device 7, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on electronic device 7. Furthermore, memory 703 may include both internal and external storage units of electronic device 7. Memory 703 is used to store operating system, application programs, boot loader, data, and other programs, such as program code of computer programs. Memory 703 may also be used to temporarily store data that has been output or will be output.

[0170] If the integrated units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, when implementing all or part of the processes in the methods of the above embodiments, this application can use a computer program to instruct related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps applied in the method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include at least: any entity or device capable of carrying computer program code to a computing device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, such as a USB flash drive, a portable hard drive, a magnetic disk, or an optical disk.

[0171] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various method embodiments described above.

[0172] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the steps described in the various method embodiments above.

[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0175] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. The device / electronic device embodiments described above are merely illustrative, and the division of modules or units described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or photovoltaic modules may be combined or integrated into another system, and some features may be ignored. Furthermore, the indirect coupling, direct coupling, or communication connection shown or discussed between them can be through some interfaces, devices, or units, and can be electrical, mechanical, or other forms.

[0176] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the above embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A design method for an LSTM input fusion structure, characterized in that, This method is applicable to GNSS SoC chip systems with CNN-LSTM joint inference paths, and is applied to an input fusion structure configuration controller, wherein the input fusion structure configuration controller is connected to a main control CPU or a configuration engine; the method includes: The system acquires the feature frames for the current task, which is a GNSS sensing task. The feature frames include CNN-derived frames and navigation system native sliding window frames. The CNN-derived frames are two-dimensional feature maps output by the CNN module. These two-dimensional feature maps are features extracted from the power spectrum or residual map of the GNSS input signal based on operators. The two-dimensional feature maps include channel spectrum maps or pseudorange sliding window residual maps of the GNSS input signal. The navigation system native sliding window frames are GNSS sliding window feature maps, which are time-frequency or time-channel observation arrays constructed through receiver front-end, correlators, or frequency domain processing. Based on a preset channel mapping method, the feature frame is expanded into a sliding window sequence according to the time dimension; this step includes: based on the preset channel mapping method, expanding the two-dimensional data in the two-dimensional feature map or the GNSS sliding window feature map into the sliding window sequence according to the time dimension; the two-dimensional data includes: an output channel and a time step, the output channel contains the features of the two-dimensional feature map or the GNSS sliding window feature map; the sliding window sequence serves as the input sequence for the current moment; Based on the data concatenation rules between the current input sequence and the hidden state from the previous time step, the target LSTM input vector for the current task is constructed; this step includes: The current input sequence is concatenated or arranged side-by-side with the hidden state from the previous time step, and the concatenated or arranged result is used as the target LSTM input vector for the current task; or The current input sequence is used as the target LSTM input vector for the current task; the current input sequence is not concatenated or arranged side by side with the hidden state of the previous time step.

2. The LSTM input fusion structure design method according to claim 1, characterized in that, The preset channel mapping method includes: a first mapping method; based on the preset channel mapping method, the feature frames are expanded into a sliding window sequence according to the time dimension, including: Based on the sliding window frame order or channel interleaving method, each CNN output channel is concatenated with each time step to obtain the sliding window sequence; Each CNN output channel is mapped to the sliding window sequence.

3. The LSTM input fusion structure design method according to claim 1, characterized in that, The preset channel mapping method includes: a second mapping method; based on the preset channel mapping method, the feature frames are expanded into a sliding window sequence according to the time dimension, including: Load the valid CNN output channels based on the channel mask configured for the current task; Based on the sliding window frame order or channel interleaving method, the effective CNN output channels are concatenated with each time step to obtain the sliding window sequence; The effective CNN output channels are mapped to the sliding window sequence.

4. The LSTM input fusion structure design method according to claim 3, characterized in that, The method further includes: The data bit width of the effective CNN output channel is compressed according to the preset compression precision mode.

5. The LSTM input fusion structure design method according to claim 4, characterized in that, The preset compression precision modes include: INT8 mode and INT4 mode; according to the preset compression precision modes, the data bit width of the effective CNN output channels is compressed, including: According to the INT8 mode, the data bit width of the effective CNN output channel is compressed to INT8; Alternatively, according to the INT4 mode, the data bit width of the effective CNN output channel can be compressed to INT4.

6. The LSTM input fusion structure design method according to any one of claims 1-5, characterized in that, The method further includes: Configure the sliding window size, channel mask, input shape transformation method, data bit width, and activation flag based on the current task number.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the LSTM input fusion structure design method according to any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the LSTM input fusion structure design method according to any one of claims 1-6.

Images