A multi-modal device and a storage resource calling method, configuration method and device thereof

Abstract

This application discloses a multimodal device and its storage resource mobilization method, configuration method, and apparatus, applied in the field of image processing technology. The method includes: receiving a mode selection signal; dynamically switching the logical connection relationship of physical storage sub-units between a first operating mode and a second operating mode based on the mode selection signal; and mobilizing at least one physical storage sub-unit according to the logical connection relationship to achieve time-division multiplexing of the physical storage sub-units to serve either the first or second operating mode. This application centrally manages physical storage resources through a central management module and dynamically reconstructs the logical connection of storage units based on the mode selection signal, solving the problem of resource idleness and waste caused by physical isolation or maximum configuration of storage resources in existing dual-modal devices, achieving a significant reduction in device area cost and improved storage resource utilization.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to multimodal devices and their storage resource access methods, configuration methods and apparatus. Background Technology

[0002] In existing dual-modal (visible light and thermal imaging) or multi-modal (visible light and thermal imaging with other imaging modes) image processing device designs, independent storage architectures are typically used to meet the image data processing requirements of different imaging modes. Taking visible light and thermal imaging dual-modal as an example, the system designs an independent set of storage resources for the infrared thermal imaging mode, including line cache, DDR interface, and on-chip SRAM, while designing another independent set of storage resources for the visible light imaging mode. These two storage systems are physically isolated within the device and do not interfere with each other.

[0003] However, in practical applications, visible light imaging mode and infrared thermal imaging mode are usually time-sharing or mutually exclusive. When the device operates in infrared mode, a large amount of storage resources allocated for visible light mode are idle, and vice versa. If, in order to simplify the design, all storage resources are allocated uniformly according to the higher specification requirement of the two modes (usually the high-resolution visible light mode), it will result in a significant waste of storage resources in the low-resolution infrared mode.

[0004] Therefore, this independent or maximum-specification storage architecture inevitably leads to problems such as high device area costs, low resource utilization, and high power consumption. How to effectively improve the utilization of on-chip storage resources and avoid wasting area and power consumption while meeting the performance requirements of dual-modal or multi-modal image processing has become an urgent technical challenge. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal device and its storage resource retrieval method, configuration method, and apparatus to solve the technical problem of storage resource waste in existing technologies under dual-modal or multimodal conditions. The specific technical solution is as follows: In a first aspect, embodiments of the present invention provide a multi-mode device, comprising: a central management module, configured to receive a mode selection signal and dynamically switch the logical connection relationship of physical storage sub-units between a first working mode and a second working mode according to the mode selection signal, wherein the first working mode and the second working mode have different physical storage resource requirements, the mode selection signal is used to indicate whether to work in the first working mode or in the second working mode, and each physical storage sub-unit is configured to be invoked in both the first working mode and the second working mode; and a physical storage unit, comprising multiple physical storage sub-units, at least one of which is time-division multiplexed to serve the first working mode or the second working mode according to the logical connection relationship, wherein two or more invoked physical storage sub-units are combined into a storage unit that meets the requirements of the working mode selected by the mode selection signal by means of bit-width splicing and / or depth splicing.

[0006] As an example, the central management module selects two or more physical storage sub-units for logical concatenation to meet the physical storage resource requirements under the current working mode. The logical concatenation includes at least one of the following methods: bit-width concatenation, which combines at least two physical storage sub-units in the bit-width dimension, such that the bit-width of the concatenated unit is not less than the bit-width requirement of the current working mode; depth concatenation, which combines at least two physical storage sub-units in the depth dimension, such that the depth of the concatenated unit is not less than the depth requirement of the current working mode; and hybrid concatenation, which combines at least two physical storage sub-units simultaneously in both the bit-width and depth dimensions, such that both the bit-width and depth of the concatenated unit are not less than the corresponding requirements of the current working mode.

[0007] As an example, the bit width and / or depth specifications of some physical storage sub-cells are configured to be a common multiple of the maximum storage cell requirements of the first operating mode and the second operating mode, so that a single physical storage sub-cell can meet the requirements of the two modes respectively through time-division multiplexing.

[0008] As an example, the bit width and / or depth specifications of some physical storage sub-cells are configured to serve the current operating mode, either alone or in combination with other physical storage sub-cells.

[0009] As one embodiment, the multimodal device further includes: a first modal core, used to operate in a first operating mode based on the physical storage sub-units connected by logical connections; and a second modal core, used to operate in a second operating mode based on the physical storage sub-units connected by logical connections.

[0010] As an example, the multi-mode device has a preset address mapping logic for physical storage sub-units, which is used to directly activate the corresponding physical storage sub-unit combination path according to the mode selection signal during the operation phase. Through bit width splicing and / or depth splicing, multiple physical storage sub-units are combined into a storage unit that meets the requirements of the current working mode.

[0011] Secondly, embodiments of the present invention provide a method for retrieving storage resources for a multi-mode device, comprising: receiving a mode selection signal, the mode selection signal indicating whether to operate in a first operating mode or a second operating mode; dynamically switching the logical connection relationship of physical storage sub-units between the first operating mode and the second operating mode according to the mode selection signal, wherein the first operating mode and the second operating mode have different physical storage resource requirements, and each physical storage sub-unit is configured to be invoked in both the first operating mode and the second operating mode; invoking at least one physical storage sub-unit in the physical storage units according to the logical connection relationship, so as to realize time-division multiplexing of physical storage sub-units to serve the first operating mode or the second operating mode, wherein two or more invoked physical storage sub-units are combined into a storage unit that meets the requirements of the operating mode selected by the mode selection signal by means of bit-width splicing and / or depth splicing.

[0012] As an example, dynamically switching the logical connection relationship of physical storage sub-units between a first operating mode and a second operating mode based on the mode selection signal includes: selecting two or more physical storage sub-units for logical concatenation to meet the physical storage resource requirements under the current operating mode; the logical concatenation includes at least one of the following methods: bit-width concatenation, combining at least two physical storage sub-units in the bit-width dimension such that the bit-width of the concatenated unit is not lower than the bit-width requirement of the current operating mode; depth concatenation, combining at least two physical storage sub-units in the depth dimension such that the depth of the concatenated unit is not lower than the depth requirement of the current operating mode; and hybrid concatenation, combining at least two physical storage sub-units simultaneously in both the bit-width and depth dimensions such that both the bit-width and depth of the concatenated unit are not lower than the corresponding requirements of the current operating mode.

[0013] As an example, the logical connection relationship of dynamically switching physical storage sub-units between the first working mode and the second working mode according to the mode selection signal includes: calling the same physical storage sub-unit in a time-division multiplexing manner in the first working mode and the second working mode to meet the storage requirements of the mode; wherein, the bit width and / or depth specification of the physical storage sub-unit is a common multiple of the maximum storage unit requirements of the first working mode and the second working mode.

[0014] As one embodiment, calling at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship includes: calling a physical storage sub-unit alone to meet the storage requirements of the mode; or calling the physical storage sub-unit in combination with other physical storage sub-units to meet the storage requirements of the mode.

[0015] As one embodiment, calling at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship includes: during the operation phase, calling a preset address mapping logic according to the mode selection signal, the address mapping logic including the physical storage sub-unit to be called and its address; directly activating the corresponding physical storage sub-unit combination path according to the address mapping logic; and dynamically combining multiple physical storage sub-units into a storage unit that meets the requirements of the current working mode through bit width splicing and / or depth splicing.

[0016] Thirdly, embodiments of the present invention provide a method for configuring storage resources in a multi-mode device, comprising: obtaining the physical storage resource requirements of a first operating mode and a second operating mode, wherein the requirements include bit width requirements and depth requirements; dividing a physical storage unit into multiple physical storage sub-units according to the requirements, and configuring the bit width and / or depth specifications of each physical storage sub-unit, wherein each physical storage sub-unit is configured to be invoked in both the first and second operating modes; establishing logical connection relationships between the physical storage sub-units and the first and second operating modes, so that at least one physical storage sub-unit can be time-division multiplexed in different operating modes, wherein two or more invoked physical storage sub-units are combined into a storage unit that satisfies the requirements of the operating mode selected by the mode selection signal by bit width splicing and / or depth splicing.

[0017] As one embodiment, establishing a logical connection relationship includes: configuring a logical splicing relationship between two or more physical storage sub-units to meet the physical storage resource requirements of the current working mode; the logical splicing relationship includes at least one of the following methods: configuring a bit-width splicing relationship, combining at least two physical storage sub-units in the bit-width dimension, such that the bit-width of the spliced ​​result is not lower than the bit-width requirement of the current working mode; configuring a depth splicing relationship, combining at least two physical storage sub-units in the depth dimension, such that the depth of the spliced ​​result is not lower than the depth requirement of the current working mode; configuring a hybrid splicing relationship, combining at least two physical storage sub-units simultaneously in the bit-width and depth dimensions, such that both the bit-width and depth of the spliced ​​result are not lower than the corresponding requirements of the current working mode.

[0018] As one embodiment, configuring the bit width and / or depth specifications of each physical storage sub-unit includes: analyzing the maximum image width supported by the first working mode and the second working mode respectively, and finding a common multiple relationship between the maximum image widths; when a common multiple relationship exists, configuring the bit width and / or depth specifications of a physical storage sub-unit as a common multiple, so that the physical storage sub-unit can be used in both working modes.

[0019] As one embodiment, establishing the logical connection between physical storage sub-units and the first and second working modes includes: generating a first storage requirement list for the first working mode and a second storage requirement list for the second working mode; performing a greedy match between unmatched storage requirement items in the first storage requirement list and storage requirement items in the second storage requirement list; wherein the greedy match includes at least one of the following methods: one-to-one matching, directly pairing one item in the first storage requirement list with another item in the second storage requirement list that meets both the depth and bit width requirements; many-to-one splicing matching, splicing multiple storage requirement items in the second storage requirement list in the bit width dimension and / or depth dimension, and pairing them with one item in the first storage requirement list; one-to-many splitting matching, splitting a large storage requirement item in the first storage requirement list into multiple sub-requirements, and pairing them with multiple storage requirement items in the second storage requirement list respectively.

[0020] As an example, before the greedy matching is executed, the method further includes: sorting the first storage requirement list in descending order of depth and / or bit width; sorting the second storage requirement list in ascending order of depth and / or bit width; wherein, a one-to-one matching traverses the sorted first storage requirement list, and searches for the smallest storage requirement item in the sorted second storage requirement list whose depth and bit width are both greater than or equal to the current item for pairing.

[0021] As an example, after establishing the logical connection relationship, the method further includes: solidifying the logical connection relationship into a preset address mapping logic, so that during the operation phase, the corresponding physical storage sub-unit combination path is directly activated according to the mode selection signal, and multiple physical storage sub-units are dynamically combined into a storage unit that meets the requirements of the current working mode through bit width splicing and / or depth splicing.

[0022] Fourthly, embodiments of the present invention provide a multimodal storage resource mobilization device, comprising: a central management module for receiving a mode selection signal, the mode selection signal indicating whether to operate in a first working mode or a second working mode; a storage switching module for dynamically switching the logical connection relationship of physical storage sub-units between the first working mode and the second working mode according to the mode selection signal, wherein the first working mode and the second working mode have different physical storage resource requirements, and each physical storage sub-unit is configured to be mobilized in both the first working mode and the second working mode; and a storage multiplexing module for mobilizing at least one physical storage sub-unit among the physical storage units according to the logical connection relationship, so as to realize time-division multiplexing of physical storage sub-units to serve the first working mode or the second working mode, wherein two or more mobilized physical storage sub-units are combined into a storage unit that meets the requirements of the working mode selected by the mode selection signal by bit-width splicing and / or depth splicing.

[0023] Fifthly, embodiments of the present invention provide an electronic device, comprising: a memory for storing a computer program; and a processor for executing the program stored in the memory to implement the method described in either the second or third aspect.

[0024] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in either the second or third aspect.

[0025] This invention provides a multimodal device and its storage resource retrieval method, configuration method, and apparatus. By establishing a central management module to uniformly manage physical storage resources and dynamically reconfiguring the logical connection relationship of storage units based on mode selection signals, and by adopting a strategy combining cutting and splicing, it solves the problem of resource idleness and waste caused by physical isolation of storage resources or configuration at the maximum specification in existing dual-modal devices, and achieves the effects of significantly reducing device area cost, improving storage resource utilization, and reducing system power consumption.

[0026] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0028] Figure 1A comparison diagram of the architectures of traditional independent storage solutions and the reused storage solution of this application; Figure 2 This is a diagram of the internal architecture of the MISC module provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a multimodal device provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the logical splicing method of physical storage sub-units provided in the embodiments of this application; Figure 5 This is a flowchart illustrating a method for accessing storage resources of a multimodal device, as provided in an embodiment of this application. Figure 6 This is a flowchart illustrating a storage resource reuse method in IR mode according to an embodiment of this application. Figure 7 This is a flowchart illustrating a storage resource reuse method in ISP mode according to an embodiment of this application. Figure 8 A flowchart illustrating a storage resource configuration method for a multimodal device provided in an embodiment of this application; Figure 9 A flowchart illustrating a configuration method provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a multimodal storage resource retrieval device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0030] Terminology Explanation: IR (Infrared): Infrared thermal imaging mode, refers to the technical scenario of imaging using thermal radiation.

[0031] ISP (Image): Visible light imaging mode, refers to the technical scenario of imaging using visible light sensors.

[0032] IR-ISP dual-mode: refers to the state in which the same device or system supports switching or time-division operation between infrared thermal imaging mode and visible light imaging mode.

[0033] SRAM reuse refers to the dynamic configuration at the hardware level to allow the same physical storage resource (SRAM) to serve different image processing modules in different working modes, or to be shared by modules with different resolution requirements in the same mode.

[0034] DDR (Double Data Rate): Double Data Rate Synchronous Dynamic Random Access Memory, is a mainstream interface standard for DRAM (Dynamic Random Access Memory). Unlike on-chip SRAM, DDR has the advantages of high storage density and low unit cost, but its access latency is relatively high.

[0035] Lnbuf (Line Buffer): A line buffer used to store one or more lines of image data. It is commonly used in algorithms such as image scaling and noise reduction.

[0036] TSP / ISP: These refer to the Infrared Image Processing Pipeline (Thermal Signal Processor) and the Visible Light Image Processing Pipeline (Image Signal Processor), respectively.

[0037] MISC module: The central management module, which serves as the core of multiplexing control in this solution, is responsible for interface selection, storage management, and algorithm scheduling.

[0038] Cutting: Logically dividing a large physical memory block into multiple smaller storage units to meet the fragmentation needs of different resolutions or different modules.

[0039] Splicing: Logically combining multiple small physical memory blocks into a large storage unit to meet the needs of high resolution or large data volume.

[0040] Redundant bits: Unused storage space reserved during storage resource reuse because the specifications cannot be perfectly matched (such as depth or width not being divisible).

[0041] This application provides a multimodal device and its storage resource access method, configuration method, and apparatus, applicable to the field of image processing technology. This solution aims to address the technical problems of low resource utilization, high device area, and high power consumption caused by the independent physical storage architecture of existing dual-modal or multimodal devices. It should be noted that the multimodal device described in this application can be a multimodal integrated circuit device.

[0042] The core idea of ​​this application is to break down the physical isolation of storage between different imaging modules by setting a MISC module (central management module) at the top level of the device to uniformly instantiate and manage the cache resources required by different imaging modules. By dynamically switching the logical ownership and connection relationship of storage resources through mode selection signals, the reuse effect of "one set of physical storage, two logical uses" is achieved.

[0043] For ease of explanation, the following will use an IR and ISP dual-mode device as an exemplary embodiment to describe the technical solution of this application in detail. It should be noted that the scope of protection of this application is not limited thereto. The application methods are similar for other forms of dual-mode devices such as IR and X-ray, ISP and X-ray, and will not be elaborated upon further. Similarly, when the device integrates three or more imaging modes such as X-ray, the same solution can be used.

[0044] Figure 1 This is a comparison diagram of the architecture of the traditional independent storage solution and the reused storage solution of this application. Figure 1 On the left is a traditional standalone SoC device solution, where the IR and ISP each have their own independent storage systems and do not interfere with each other, but resources are seriously idle. Figure 1 The right side shows the multimodal SoC device solution proposed in this application. It uses a central management module (MISC) to uniformly manage a shared memory resource pool. The MISC has three sub-units: DDR, SRAM, and algorithms. This application is a technical solution for shared SRAM storage resources. The IR core and ISP core access this resource pool through a unified interface, dynamically meeting their respective needs. The multimodal device also includes: a first-mode core, used in the first operating mode to operate based on the physical storage sub-units connected by logical connections; and a second-mode core, used in the second operating mode to operate based on the physical storage sub-units connected by logical connections. The first-mode core is the thermal imaging IR core, and the second-mode core is the visible light ISP core. The visible light ISP core is responsible for processing raw image data from a visible light sensor. The ISP core outputs standard format image data that meets the requirements of display or subsequent visual analysis for use by the downstream "image display" module. The thermal imaging IR core is responsible for processing raw infrared data from a thermal imaging sensor. The IR core ultimately outputs a visualized image with temperature information or thermal distribution characteristics for presentation by the "image display" module.

[0045] Although the ISP core and IR core are functionally independent, they can share resources in this architecture through "reused components": when the system only processes visible light images, the IR core is idle, and its available resources (such as SRAM, DDR, and algorithm modules) can be borrowed by the ISP core to improve performance; the reverse is also true. In dual-modal synchronous processing scenarios, the ISP core and IR core can run simultaneously, and the reused components act as a "resource pool" to dynamically allocate computing power and bandwidth. The mode selection signal is not only used for path selection, but can also trigger resource reconfiguration logic to ensure that the system always runs efficiently.

[0046] See Figure 2 The diagram shows the storage resource reuse architecture, illustrating the detailed architecture within the central management module MISC. The physical storage resource pool contains uniformly instantiated reused storage resources, including reused SRAM (e.g., shared storage block A, shared storage block B, and shared storage block C). Through dynamic routing selection logic, the input / output ports are mapped to the IR processing module or ISP processing module based on the mode selection signal, while remaining transparent or disconnected from modules that are not selected.

[0047] Figure 3 This is a schematic diagram of a multimodal device according to an embodiment of this application. For example... Figure 3 As shown, the multimodal device includes: The central management module 30 is used to receive the mode selection signal and dynamically switch the logical connection relationship of the physical storage sub-units between the first working mode and the second working mode according to the mode selection signal. The first working mode and the second working mode have different physical storage resource requirements. The mode selection signal is used to indicate whether to work in the first working mode or in the second working mode. Each physical storage sub-unit is configured to be invoked in both the first working mode and the second working mode.

[0048] The first operating mode is the IR processing mode for IR processing, and the second operating mode is the ISP processing mode for ISP processing. The mode selection signal indicates whether the current operating mode is the IR processing module or the ISP processing module. The central management module selects the corresponding physical storage resource based on the operating mode selected by the mode selection signal. Specifically, upon receiving the mode selection signal, when the mode selection signal indicates IR mode, the logic inside the MISC module maps the output / input ports of the physical storage sub-unit to the interface of the IR module, while simultaneously shielding or disconnecting the connection with the ISP; when the mode selection signal indicates ISP mode, the same physical storage sub-unit is mapped to the ISP module.

[0049] Each physical storage sub-unit can be invoked in either the first or second working mode. This achieves time-sharing multiplexing of the same physical storage sub-unit under different working modes, improving the utilization rate of physical storage sub-units and avoiding resource waste during idle periods.

[0050] The logical connection relationship causes the total area of ​​the configured physical storage sub-units to expand relative to the larger area of ​​the independently configured storage units in the first and second working modes, with an area expansion rate of less than 5%. The method for determining an area expansion rate of less than 5% is as follows: if the storage area required for the independently configured first working mode is A, the storage area required for the independently configured second working mode is B, and the total area of ​​the configured physical storage sub-units is C; then the area expansion rate = [C - max(A,B)] / max(A,B) × 100%, and this area expansion rate is less than 5%.

[0051] The physical storage unit 32 includes multiple physical storage sub-units. At least one physical storage sub-unit is time-division multiplexed to serve the first working mode or the second working mode according to the logical connection relationship. In this way, two or more physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal by bit width splicing and / or depth splicing.

[0052] Configure a physical storage unit as multiple physical storage sub-units (e.g.) Figure 2 The collection of shared storage blocks makes it easier to arrange and combine physical storage sub-units in different working modes to meet the physical storage resource requirements of different working modes. For example, by combining physical storage sub-units of different specifications in the depth and / or bit width dimensions, the physical storage resource requirements of the first working mode and the second working mode can be met respectively.

[0053] This application achieves the technical effect of reusing storage resources by dividing the physical storage unit into a set of multiple physical storage sub-units and using a combination and splicing method to reuse these sub-units. This avoids the waste of resources caused by the storage unit in another working mode being idle when two independent sets of physical storage units are used in a single working mode.

[0054] Optionally, the central management module selects two or more physical storage sub-units for logical concatenation to meet the physical storage resource requirements under the current working mode. The logical concatenation includes at least one of the following methods: bit-width concatenation, which combines at least two physical storage sub-units in the bit-width dimension, such that the bit-width of the concatenated unit is not less than the bit-width requirement of the current working mode; depth concatenation, which combines at least two physical storage sub-units in the depth dimension, such that the depth of the concatenated unit is not less than the depth requirement of the current working mode; and hybrid concatenation, which combines at least two physical storage sub-units simultaneously in both the bit-width and depth dimensions, such that both the bit-width and depth of the concatenated unit are not less than the corresponding requirements of the current working mode.

[0055] Figure 4 The illustration shows the logical splicing method of physical storage sub-units in an embodiment of this application. When the bit width required by the IR / IMG is greater than the bit width of a single physical storage sub-unit, multiple narrow-bit-width physical storage sub-units are spliced ​​horizontally; when the depth required by the IR is greater than the depth of a single physical storage sub-unit, multiple shallow-depth physical storage sub-units are spliced ​​vertically.

[0056] (a) Depth stitching scheme 1. Deep stitching in ISP mode In ISP mode, the original requirement includes two processing modules, each with a depth of 1440 units × 12b / 10b data bit width. Using a depth-stitching strategy, the two instantiated entities (i.e., physical storage sub-units) are stacked vertically along the processing depth direction to form a unified multiplexed module with a height of 2880 units and a bit width of 12b (the larger value). This solution is suitable for scenarios requiring increased throughput but with a fixed bit width, such as parallel processing of high-speed video streams.

[0057] 2. Non-stitching preservation in IR mode In IR mode, the original requirement is three independent modules with a depth of 960 units and a width of 12 bits. The instantiated entity meets the specifications of three independent modules; therefore, this embodiment does not perform a splicing operation, maintaining the original structure unchanged. That is, the IR multiplexing module remains three independent modules and does not participate in depth splicing. In other words, the depth splicing strategy is not mandatory for all modules and can be selectively enabled according to actual needs.

[0058] (b) Bit width + depth splicing scheme 1. Heterogeneous resource splicing under ISP model In ISP mode, the original requirements included three modules with different bit widths: 4b×512, 8b×512, and 10b×256. When using a bit width + depth stitching strategy, the three instantiated entities (i.e., physical storage sub-units) are logically invoked as independent units. Each module maintains its original bit width and depth, but storage resources are reused through shared control logic or a bus interface. This solution is suitable for composite image scenarios that require simultaneous processing of multiple resolutions or color depths.

[0059] 2. Deep decomposition and reuse in IR mode In IR mode, the original requirement is a single module with a depth of 800 units and a width of 14 bits. When using a bit width + depth concatenation strategy, three instantiated entities with different bit widths and depths (4b×512, 8b×512, and 10b×256) are logically concatenated into a storage unit with a depth greater than 800 units and a width of 14 bits, and then scheduled and used on demand in the system.

[0060] (c) Width splicing scheme 1. Bit width extension in ISP mode In ISP mode, the original requirement was a module with a depth of 720 units and a bit width of 54 bits. Using a bit-width splicing strategy, two instantiated entities with a depth of 800 units and a bit width of 28 bits are horizontally spliced ​​together to form a unified multiplexed module with a total bit width of 56 bits (28 bits + 28 bits) and a depth of 800 units. This solution is suitable for scenarios requiring higher data bandwidth but sensitive to latency, such as real-time wide-format image enhancement.

[0061] 2. No-stitching preservation in IR mode In IR mode, the original requirement is two independent modules with a depth of 800 units and a width of 28 bits. The instantiated entity meets the specification requirements of two independent modules. In this embodiment, the bit width is not spliced, and the original structure remains unchanged. That is, the IR multiplexing module is still two independent modules and does not participate in horizontal splicing. The bit width splicing strategy can also be selectively enabled according to the module characteristics to avoid unnecessary resource waste.

[0062] This invention introduces three flexible storage resource reuse strategies: depth splicing, bit width splicing, and bit width + depth splicing. The spliced ​​storage resource rules are greater than or equal to the specification requirements of the original working mode. Furthermore, through resource reuse and splicing, the number of redundant logic gates is reduced, splicing modules are activated on demand, all modules are avoided from running, storage resource utilization is improved, and resource waste caused by idleness is avoided.

[0063] The bit width and / or depth specifications of some physical storage sub-cells are configured to be a common multiple of the maximum storage cell requirements of the first operating mode and the second operating mode, so that a single physical storage sub-cell can meet the requirements of the two modes respectively through time-division multiplexing.

[0064] In a specific application scenario, assume the maximum resolution of IR mode is 640×480, and the maximum resolution of ISP mode is 1920×1080. IR mode requires a row width of 640, and ISP mode requires a row width of 1920. Analysis shows that 1440 is a common multiple of 640 and 1920. Therefore, a physical storage unit with a row width of 1440 can be divided into multiple physical storage sub-units. In IR mode, multiple small physical storage units can be used in combination or individually. In ISP mode, they are logically combined into a single large physical storage unit. This scheme introduces very few redundant bits, achieving efficient reuse of storage resources. Physical storage sub-units can be used individually in IR mode or as part of a larger physical storage unit in ISP mode, satisfying the needs of both modes through time-division multiplexing.

[0065] The bit width and / or depth specifications of some physical storage sub-cells are configured to serve the current operating mode, either individually or in combination with other physical storage sub-cells.

[0066] To address the differences in physical storage cell depth and width requirements caused by the different maximum resolutions of IR and ISP modes (ISP resolution is usually higher than IR), when configuring physical storage sub-cells, the bit width and / or depth specifications of some physical storage sub-cells are configured to serve the current operating mode individually or in combination with other physical storage sub-cells.

[0067] For example Figure 4 In IR mode, the original requirement was three independent modules, each 960 units deep and 12 bits wide. The instantiated entity meets the specifications of the three independent modules and is therefore configured to serve independently in IR mode, maintaining the original structure.

[0068] For example Figure 4 In ISP mode, the original requirements consist of two processing modules, each with a depth of 1440 units × 12b / 10b data width. Using a depth-stack strategy, the two instantiated entities (i.e., physical storage sub-units) are stacked vertically along the processing depth direction to form a unified multiplexed module with a height of 2880 units and a bit width of 12b (whichever is larger). These two instantiated entities are configured to combine with other physical storage sub-units to serve ISP mode.

[0069] Optionally, the multi-mode device has a preset address mapping logic for physical storage sub-units, which is used to directly activate the corresponding physical storage sub-unit combination path according to the mode selection signal during the operation phase, and combine multiple physical storage sub-units into a storage unit that meets the requirements of the current working mode through bit width splicing and / or depth splicing.

[0070] The aforementioned optimal splicing strategy is hardwired into static logical connections or pre-configured mapping tables within the device. Multimodal devices have pre-defined splicing strategies for various physical storage sub-units and corresponding address mapping logic. During actual device operation, when switching to IR or IMG scenarios via a mode selection signal, the central management module MISC does not need to perform real-time resource lookups or algorithm calculations. Instead, it directly activates the corresponding physical block combination path instantly based on the pre-defined mapping logic. If the logic cache required by the algorithm module in the current scenario is larger than the size of a single physical storage sub-unit, the central management module MISC will automatically call the hardwired splicing strategy to seamlessly reassemble the selected multiple physical storage sub-units into a continuous virtual large storage unit for transparent access by upper-layer algorithms. This mechanism ensures optimal resource utilization while avoiding the timing overhead of dynamic scheduling at runtime.

[0071] This application also provides a method for accessing the storage resources of a multimodal device. For example... Figure 5 As shown, the method for accessing the storage resources of this multimode device includes the following steps: Step S501, mode selection signal, the mode selection signal is used to indicate whether to work in the first working mode or in the second working mode.

[0072] The central management module (MISC) receives a mode selection signal, which indicates whether the system is currently operating in the first working mode (infrared thermal imaging mode (IR mode)) or the second working mode (visible light imaging mode (ISP mode)). The mode selection signal can be configured by the upper-level software of the system or automatically generated by the hardware after detecting the current imaging mode.

[0073] Step S503: Dynamically switch the logical connection relationship of physical storage sub-units between the first working mode and the second working mode according to the mode selection signal. The first working mode and the second working mode have different physical storage resource requirements. Each physical storage sub-unit is configured to be invoked in both the first working mode and the second working mode.

[0074] Each physical storage sub-unit can be invoked in either the first or second working mode. This achieves time-sharing multiplexing of the same physical storage sub-unit under different working modes, improving the utilization rate of physical storage sub-units and avoiding resource waste during idle periods.

[0075] When the mode selection signal indicates IR mode, the internal logic of the MISC module maps the output / input ports of the physical storage sub-unit to the interface of the IR processing module, while simultaneously masking or disconnecting the connection with the ISP processing module. When the mode selection signal indicates ISP mode, the same physical storage sub-unit is mapped to the ISP processing module. IR mode and ISP mode have different physical storage resource requirements; for example, ISP mode typically requires greater depth and bit width to support high-resolution image processing.

[0076] Step S505: At least one physical storage sub-unit in the physical storage unit is called according to the logical connection relationship, so as to realize the time-division multiplexing of the physical storage sub-unit to serve the first working mode or the second working mode. In this step, two or more physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal by bit width splicing and / or depth splicing.

[0077] When a logical connection indicates the invocation of a single physical storage subunit, that subunit is used exclusively by the current operating mode. When a logical connection indicates the invocation of multiple physical storage subunits, these subunits are combined for use by the current operating mode. Physical storage subunits can be used individually in IR mode or as part of a physical storage resource in ISP mode, satisfying the needs of both modes through time-sharing multiplexing.

[0078] This application embodiment solves the technical problem of low utilization of storage resources on devices in the prior art by dynamically switching the logical connection relationship of physical storage sub-units between the first working mode and the second working mode, and by calling one or more physical storage sub-units in a time-division multiplexing manner to meet the needs of the two modes respectively, thereby achieving the technical effect of improving the utilization of storage resources.

[0079] Optionally, dynamically switching the logical connection relationship of physical storage sub-units between the first and second operating modes based on the mode selection signal includes: selecting two or more physical storage sub-units for logical concatenation to meet the physical storage resource requirements under the current operating mode; the logical concatenation includes at least one of the following methods: bit-width concatenation, combining at least two physical storage sub-units in the bit-width dimension, such that the bit-width of the concatenated unit is not lower than the bit-width requirement of the current operating mode; depth concatenation, combining at least two physical storage sub-units in the depth dimension, such that the depth of the concatenated unit is not lower than the depth requirement of the current operating mode; and hybrid concatenation, combining at least two physical storage sub-units simultaneously in both the bit-width and depth dimensions, such that both the bit-width and depth of the concatenated unit are not lower than the corresponding requirements of the current operating mode.

[0080] As an embodiment of the above method, when dynamically switching logical connection relationships, the central management module can select two or more physical storage sub-units for logical splicing to meet the physical storage resource requirements under the current working mode.

[0081] (1) Width splicing See Figure 4 In section (c), when the bit width required by the IR / ISP module is greater than the bit width of a single physical storage sub-cell, multiple narrow-bit-width sub-cells are horizontally spliced ​​together. For example, in ISP mode, a storage cell with a depth of 720 units and a bit width of 54 bits is required, while the instantiated physical storage sub-cell has a depth of 800 units and a bit width of 28 bits. In this case, two 28-bit-width sub-cells are horizontally spliced ​​together to form a unified multiplexed module with a total bit width of 56 bits (greater than 54 bits) and a depth of 800 units, thus meeting the requirements of the ISP module.

[0082] (2) Depth splicing See Figure 4 In (a), when the required depth of the IR / ISP module is greater than the depth of a single physical storage sub-unit, multiple shallow-depth sub-units are vertically stacked. For example, in ISP mode, two storage modules with a depth of 1440 units × 12b / 10b are required. The two 1440-unit deep sub-units are stacked vertically along the depth direction to form a unified multiplexed module with a depth of 2880 units and a larger bit width (12b).

[0083] (3) Mixed splicing See Figure 4 In section (b), when it is necessary to combine components simultaneously in both the bit width and depth dimensions, a hybrid splicing method is used. For example, in IR mode, a storage module with a depth of 800 units and a bit width of 14 bits is required. Three sub-cells of different specifications (4b×512, 8b×512, and 10b×256) are spliced ​​and combined simultaneously in both the bit width and depth directions to form a storage cell that meets the requirements.

[0084] Optionally, the logical connection relationship of the physical storage sub-units dynamically switched between the first working mode and the second working mode according to the mode selection signal includes: calling the same physical storage sub-unit in a time-division multiplexing manner in the first working mode and the second working mode to meet the storage requirements of the mode; wherein the bit width and / or depth specification of the physical storage sub-unit is a common multiple of the maximum storage unit requirements of the first working mode and the second working mode.

[0085] As an embodiment of the above method, in a specific application scenario, assume that the maximum resolution of IR mode is 640×480 and the maximum resolution of ISP mode is 1920×1080. The required line width for IR mode is 640, and the required line width for ISP mode is 1920. Through analysis, 1440 is a common multiple of 640 and 1920.

[0086] Therefore, in the traditional scheme, a physical storage sub-unit with a bit width of 1440 is configured in this application as multiple small blocks (e.g., 640+640+160) of physical storage sub-units, which are used by different sub-modules of IR in IR mode; and logically combined into a whole large physical storage unit in ISP mode.

[0087] This physical storage sub-unit is used in a time-division multiplexing manner, and can be used in both working modes with a very small number of redundant bits, thus achieving efficient reuse of storage resources.

[0088] Optionally, calling at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship includes: calling a physical storage sub-unit alone to meet the storage requirements of the mode; or calling the physical storage sub-unit in combination with other physical storage sub-units to meet the storage requirements of the mode.

[0089] As an embodiment of the above method, when calling physical storage sub-units according to logical connection relationships, the following two methods can be used: Called separately in the first working mode: for example Figure 4 In the IR mode of (c), the two original requirements are independent modules with a depth of 800 units and a width of 28 bits. At this time, the two physical storage sub-units serve their respective IR sub-modules independently, without the need for splicing.

[0090] Combining calls in the second working mode: for example Figure 4 In the ISP mode (c), a module with a depth of 720 units and a width of 54 bits is required. At this time, two physical storage sub-units with a width of 28 bits and a depth of 800 units, which are used separately in the IR mode, are horizontally spliced ​​together to form a large module with a width of 56 bits for use by the ISP, which meets the ISP's specifications in terms of both width and depth.

[0091] Optionally, calling at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship includes: during the operation phase, calling a preset address mapping logic according to the mode selection signal, the address mapping logic including the physical storage sub-unit to be called and its address; directly activating the corresponding physical storage sub-unit combination path according to the address mapping logic; and dynamically combining multiple physical storage sub-units into a storage unit that meets the requirements of the current working mode through bit width splicing and / or depth splicing.

[0092] As an embodiment of the above method, during the device design phase, the optimal wiring strategy is hardwired into static logic connections or a pre-configured mapping table within the device. During the actual device operation phase, the method includes: When switching to IR or ISP scenarios via the mode selection signal, the MISC module calls the preset address mapping logic, which includes the physical storage sub-unit identifier to be called and its address information.

[0093] Based on this address mapping logic, the corresponding physical storage sub-unit combination path is directly activated. Through bit-width concatenation and / or depth concatenation, multiple physical storage sub-units are dynamically combined into a storage unit that meets the requirements of the current working mode. This mechanism ensures optimal resource utilization while avoiding the timing overhead of runtime dynamic scheduling.

[0094] Figure 6 This is a flowchart of a storage resource reuse method in IR mode according to an embodiment of the present invention. The following is in conjunction with... Figure 6 The memory reuse process of this application is described.

[0095] Mode selection: If the mode selection signal is 0, select IR mode and initiate the storage resource request (Request Memory Dir: depth Dir, width Wit) of IR module A.

[0096] Resource matching and splicing strategy: The MISC management module selects one of two splicing methods based on the request: Method 1 (Depth stitching): Select MemA and MemB for deep stitching.

[0097] The "Address Remapping: High-Bit Block Selection" logic is executed to map the high bits of the logical address to different physical modules.

[0098] The "address distribution: high-order chip select" signal is sent to MemA (bit width B1, depth D1) and MemB (bit width B2, depth D2) respectively to achieve physical support.

[0099] Method 2 (Bit Width Concatenation): Select MemC and MemD to concatenate the bit widths.

[0100] Execute the "Data width merging: multi-path parallel" logic.

[0101] Send "Data Splitting: Low-bit Broadcast" signals to MemC (bit width B3, depth D3) and MemD (bit width B4, depth D4) respectively to achieve physical support.

[0102] Result: The large-capacity physical storage sub-units required by the logic IR module A are ultimately physically supported by multiple small physical storage sub-units (MemA~D) through the two splicing methods described above.

[0103] Figure 7 This is a flowchart of a storage resource reuse method in ISP mode according to an embodiment of the present invention. The following is in conjunction with... Figure 7 The memory reuse process of this application is described.

[0104] This flowchart describes how, in ISP mode, two logical modules (ISP module A and ISP module B) use the MISC internal engine to call physical storage sub-units (MemA~D) to splice resources to meet their respective large-capacity memory requirements.

[0105] The specific process is as follows: Mode selection: If the mode selection signal is 1, select ISP mode and activate the storage resource requests of ISP module A and ISP module B.

[0106] ISP module A path: Memory request: Depth: Dispa, Width: Wispa.

[0107] MISC management module matching and splicing: Select MemA and MemC for deep splicing.

[0108] Execute "Address Remapping: High-Bit Block Selection", and access MemA (bit width B1, depth D1) and MemC (bit width B3, depth D3) respectively through "Address Distribution: High-Bit Chip Selection" to achieve physical support and build "Memory required for Logical ISP Module A".

[0109] ISP module B path: Requested memory: Depth Dispb, Wispb width.

[0110] MISC management module matching and splicing: Select MemB and MemD to splice the bit width.

[0111] The "Data Bit Width Merging: Multi-path Parallelism" is executed, and the MemB (bit width B2, depth D2) and MemD (bit width B4, depth D4) are accessed separately through "Data Splitting: Low Bit Broadcast" to achieve physical support and build the "physical storage resources required for logical ISP module B".

[0112] Result: The two logical ISP modules each call different physical memory from the physical resource pool and independently construct their required large-capacity memory structure by splicing depth or bit width.

[0113] This application also provides a method for configuring storage resources in a multi-mode device. This method is typically performed during the device design phase to determine the specifications and logical connections of physical storage sub-cells. For example... Figure 8 As shown, the storage resource configuration method for this multimode device includes the following steps: Step S802: Obtain the physical storage resource requirements of the first working mode and the second working mode respectively. The requirements include bit width requirements and depth requirements.

[0114] The requirements include bit width requirements (e.g., 28 bits in IR mode and 54 bits in ISP mode) and depth requirements (e.g., 800 units in IR mode and 1440 units in ISP mode). These requirements can be obtained by summarizing and analyzing the caching requirements of each submodule of the IR and ISP image processing modules.

[0115] Step S804: According to the requirements, the physical storage unit is divided into multiple physical storage sub-units, and the bit width and / or depth specifications of each physical storage sub-unit are configured. Each physical storage sub-unit is configured to be invoked in the first working mode and the second working mode.

[0116] Each physical storage sub-unit can be invoked in either the first or second working mode. This achieves time-sharing multiplexing of the same physical storage sub-unit under different working modes, improving the utilization rate of physical storage sub-units and avoiding resource waste during idle periods.

[0117] The physical storage unit is divided into multiple physical storage sub-units (e.g., 800 units deep × 28 bits, 1440 units deep × 12 bits, etc.), and the bit width and / or depth specifications of each physical storage sub-unit are configured. Optionally, establishing logical connection relationships includes: configuring logical splicing relationships between two or more physical storage sub-units to meet the physical storage resource requirements of the current working mode; the logical splicing relationship includes at least one of the following methods: configuring a bit width splicing relationship, combining at least two physical storage sub-units in the bit width dimension, such that the spliced ​​bit width is not less than the bit width requirement of the current working mode; configuring a depth splicing relationship, combining at least two physical storage sub-units in the depth dimension, such that the spliced ​​depth is not less than the depth requirement of the current working mode; configuring a hybrid splicing relationship, combining at least two physical storage sub-units simultaneously in the bit width and depth dimensions, such that the spliced ​​bit width and depth are both not less than the corresponding requirements of the current working mode.

[0118] Specific configuration strategies include: (1) Common multiple configuration: See the description of common multiple reuse in step S203.

[0119] (2) Bit width splicing configuration: Combine at least two physical storage sub-units in the bit width dimension so that the spliced ​​bit width is not lower than the bit width requirement of the current working mode (e.g., Figure 4 (as shown in (c)).

[0120] (3) Depth splicing configuration: Combine at least two physical storage sub-units in the depth dimension so that the depth of the spliced ​​unit is not less than the depth requirement of the current working mode (e.g., Figure 4 (as shown in (a)).

[0121] (4) Hybrid splicing configuration: At least two physical storage sub-units are combined simultaneously in both the bit width and depth dimensions, such that the bit width and depth of the spliced ​​unit are not lower than the corresponding requirements of the current working mode (e.g., Figure 4 (as shown in (b)).

[0122] Step S806: Establish logical connection relationships between physical storage sub-units and the first working mode and the second working mode, so that at least one physical storage sub-unit can be time-division multiplexed under different working modes. In this case, two or more physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal by bit width splicing and / or depth splicing.

[0123] A logical connection is established between the physical storage sub-unit and the first and second working modes, enabling at least one physical storage sub-unit to be time-division multiplexed in different working modes. Each physical storage sub-unit can be invoked in either the first or second working mode. This achieves time-division multiplexing of the same physical storage sub-unit in different working modes, improving the utilization rate of physical storage sub-units and avoiding resource waste during idle periods.

[0124] The configuration method of this application adopts a strategy of "splitting a large cache and reusing multiple small caches". During the design phase, it logically splices together multiple idle physical storage sub-units in the first working mode, combining them into a virtual large physical storage unit for use in the second working mode. Alternatively, it reverses this process, logically splitting the large physical storage unit in the second working mode into multiple physical storage sub-units in the first working mode for use in the first working mode. In summary, the completed device, by flexibly splicing multiple physical storage sub-units, can meet the needs of both the physical storage sub-units in the first working mode and the large physical storage unit in the second working mode. This reuses physical storage resources, avoiding waste, and increases the flexibility of storage resource usage.

[0125] Optionally, the bit width and / or depth specifications of each physical storage sub-unit are configured, including: analyzing the maximum image width supported by the first working mode and the second working mode respectively, and finding the common multiple relationship between the maximum image widths; when a common multiple relationship exists, the bit width and / or depth specifications of a physical storage sub-unit are configured as a common multiple, so that the physical storage sub-unit can be used in both working modes.

[0126] As an embodiment of the above method, when configuring the bit width and / or depth specifications of each physical storage sub-unit, a common multiple analysis strategy is adopted: analyze the maximum image width supported by the first working mode (such as IR) and the second working mode (such as ISP), and find the common multiple relationship between the maximum image widths. For example, the maximum row width of IR mode is 640, and the maximum row width of ISP mode is 1920, and their common multiple is 1920 (or 1440, if row and column combinations are considered).

[0127] When a common multiple relationship exists, the bit width and / or depth of a physical storage sub-cell is configured to that common multiple (e.g., 1440), so that the physical storage sub-cell can be used in both operating modes, with very few redundant bits introduced.

[0128] Optionally, establishing logical connection relationships between physical storage sub-units and the first working mode and the second working mode includes: generating a first storage requirement list for the first working mode and a second storage requirement list for the second working mode; and performing a greedy match between unmatched storage requirement items in the first storage requirement list and storage requirement items in the second storage requirement list. The greedy matching includes at least one of the following methods: one-to-one matching, where an item in the first storage requirement list meets the depth and bit width requirements in the second storage requirement list; many-to-one concatenation matching, where multiple storage requirement items in the second storage requirement list are concatenated and combined in the bit width and / or depth dimensions, and paired with an item in the first storage requirement list; one-to-many splitting matching, where a large storage requirement in the first storage requirement list is split into multiple sub-requirements, and each sub-requirement is paired with multiple storage requirement items in the second storage requirement list. Before the greedy matching is executed, the following steps are also included: sorting the first storage requirement list in descending order of depth and / or bit width; sorting the second storage requirement list in ascending order of depth and / or bit width; wherein, the one-to-one matching traverses the sorted first storage requirement list and searches for the smallest storage requirement item in the sorted second storage requirement list whose depth and bit width are both greater than or equal to the current item for pairing.

[0129] As an embodiment of the above method, a greedy matching algorithm is used when establishing logical connections. Specifically, this includes: Generate storage requirement lists for the first and second operating modes. Generate a first storage requirement list for IR mode and a second storage requirement list for ISP mode, where each item includes specifications such as depth and bit width. Sort the first storage requirement list in descending order of depth and / or bit width; sort the second storage requirement list in ascending order of depth and / or bit width. Iterate through the sorted first storage requirement list and search for matching items in the sorted second storage requirement list. Matching methods include at least one of the following: (1) One-to-one matching: Pair an item in the first storage requirement list with the smallest storage requirement item in the second storage requirement list whose depth and bit width are both greater than or equal to the current item. For example, if IR needs an 800×28 storage module, it will search for the smallest specification module with a depth ≥ 800 and a bit width ≥ 28 in the ISP list for matching. If the area difference between the two does not exceed a preset threshold (such as 5%), the match is successful.

[0130] (2) Many-to-one splicing and matching: When the IR needs a larger storage module, multiple smaller storage requirements in the ISP list are spliced ​​together in the bit width and / or depth dimensions and matched with the corresponding item in the IR list. For example, if the IR needs a 1440×12 module, two 720×12 modules in the ISP list are spliced ​​together and matched.

[0131] (3) One-to-many splitting and matching: When the IR requires multiple smaller storage modules, a large storage requirement in the ISP list is split into multiple sub-requirements, which are then paired with multiple storage requirements in the IR list. For example, the 1440×12 module in the ISP list is split to meet the three 480×12 requirements in the IR list.

[0132] Optionally, after establishing the logical connection relationship, the method further includes: solidifying the logical connection relationship into a preset address mapping logic, so that during the operation phase, the corresponding physical storage sub-unit combination path is directly activated according to the mode selection signal, and multiple physical storage sub-units are dynamically combined into a storage unit that meets the requirements of the current working mode through bit width splicing and / or depth splicing.

[0133] As an embodiment of the above method, after establishing the logical connection relationship, the method further includes solidifying the logical connection relationship into a preset address mapping logic, so that during the operation phase, the corresponding physical storage sub-unit combination path is directly activated according to the mode selection signal. Multiple physical storage sub-units are dynamically combined into a storage unit that meets the requirements of the current working mode through bit-width concatenation and / or depth concatenation.

[0134] In practical implementation, the optimal mapping logic can be written as a combinational logic circuit using a hardware description language (such as Verilog) and embedded within the MISC module. For example, the chip select signals and address offset constants of different physical memory sub-units can be preset to fixed values, and the corresponding paths can be directly activated under the drive of the mode selection signal.

[0135] This application provides a multi-mode device and its storage resource calling method, configuration method, and apparatus. It centrally manages IR and ISP dual-mode storage resources through a MISC module, dynamically reconstructs the logical ownership of physical storage sub-units through mode selection signals, and realizes dynamic path reconstruction of multi-mode data streams. Through Lnbuf bit-width adaptive splitting and splicing technology, a large-bit-width Lnbuf is split into multiple small-bit-width storage units only when the area expansion rate is below a threshold. This improves the granularity of storage resources to facilitate reuse and significantly reduces dynamic power consumption by reducing the effective read / write bit width per cycle. Through multi-dimensional physical storage sub-unit reuse technology, a bidirectional matching algorithm based on "small cache splicing for large cache" and "large cache cutting for multiple small caches" is invented. Combined with resolution common multiple analysis and storage unit splitting and splicing strategies, redundant bits in the storage unit reuse process are minimized.

[0136] Figure 9 This is a flowchart of a configuration method according to an embodiment of the present invention. The following is in conjunction with... Figure 9 The memory reuse process of this application is described.

[0137] Input and Preprocessing: Storage requirements table for input IR and ISP.

[0138] Perform preprocessing (Lnbuf splitting judgment). If the area expansion rate is lower than the threshold (the threshold is 15%), split the entry to generate a smaller bit width; otherwise, keep the original entry.

[0139] Sorting: Sort the IR list and ISP list in descending order of depth / width respectively.

[0140] Traversal matching (core strategy): Try four concatenation strategies in sequence. Once a match is found, terminate the current matching process and record the result: Strategy 1 (One-to-One Direct Matching): Check whether the requirement rules and instantiated entities are completely consistent.

[0141] Strategy 2 (Common Multiple Matching): Check if there is a multiple relationship between the depth or width.

[0142] Strategy 3 (ISP sub-item splicing and matching IR major item): Check whether storage units under multiple traditional ISP modes can be spliced ​​into a major storage unit under IR mode.

[0143] Strategy 4 (ISP Large Item Splitting and Matching IR Small Items): Check whether a large storage unit under the traditional ISP mode can be split and matched with multiple small storage units under the IR mode.

[0144] Results recording and output: If a match is successful, record the corresponding reuse relationship according to the specific strategy (e.g., reuse relationship, common multiple reuse, splicing reuse, cutting reuse).

[0145] If all strategies fail, it is marked as "exclusive resource".

[0146] Finally, an RTL code and address mapping table is generated based on all matching results.

[0147] The present invention also provides a storage resource retrieval device in a dual-modal manner. Figure 10 This is a storage resource retrieval device for a multi-modal device according to an embodiment of the present invention, such as... Figure 10 As shown, the calling device includes: The central management module 1010 is used to receive a mode selection signal, which indicates whether to operate in the first working mode or in the second working mode.

[0148] The storage switching module 1020 is used to dynamically switch the logical connection relationship of physical storage sub-units between a first working mode and a second working mode based on a mode selection signal, wherein the first working mode and the second working mode have different requirements for physical storage resources.

[0149] The storage multiplexing module 1030 calls at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship, so as to realize the time-division multiplexing of the physical storage sub-unit to serve the first working mode or the second working mode.

[0150] The central management module (MISC) receives a mode selection signal, which indicates whether the system is currently operating in the first working mode (infrared thermal imaging mode (IR mode)) or the second working mode (visible light imaging mode (ISP mode)). The mode selection signal can be configured by the upper-level software of the system or automatically generated by the hardware after detecting the current imaging mode.

[0151] When IR mode is indicated, the internal logic of the MISC module maps the output / input ports of the physical storage sub-unit to the interface of the IR processing module, while simultaneously masking or disconnecting the connection with the ISP processing module. When ISP mode is indicated, the same physical storage sub-unit is mapped to the ISP processing module. IR and ISP modes have different physical storage resource requirements; for example, ISP mode typically requires greater depth and bit width to support high-resolution image processing.

[0152] When a logical connection indicates the invocation of a single physical storage subunit, that subunit is used exclusively by the current operating mode. When a logical connection indicates the invocation of multiple physical storage subunits, these subunits are combined for use by the current operating mode. Physical storage subunits can be used individually in IR mode or as part of a physical storage resource in ISP mode, satisfying the needs of both modes through time-sharing multiplexing.

[0153] This application embodiment solves the technical problem of low utilization of storage resources on devices in the prior art by dynamically switching the logical connection relationship of physical storage sub-units between the first working mode and the second working mode, and by calling one or more physical storage sub-units in a time-division multiplexing manner to meet the needs of the two modes respectively, thereby achieving the technical effect of improving the utilization of storage resources.

[0154] The present invention also provides an electronic device, such as... Figure 11 As shown, the electronic device includes: a memory 1101 for storing computer programs; and a processor 1103 for executing any of the storage resource retrieval methods of the multimodal device when executing the program stored in the memory.

[0155] In practical applications, the storage resource retrieval method for multimodal devices provided in this application embodiment can be applied to various electronic devices, such as personal computers, servers, mobile phones, and other devices with data processing capabilities. In one implementation, to improve processing efficiency, the storage resource retrieval method for multimodal devices provided in this application embodiment can be applied to high-performance computers. Furthermore, the storage resource retrieval method for multimodal devices provided in this application embodiment can be implemented through software, hardware, or a combination of both.

[0156] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0157] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0158] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0159] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0160] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0161] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in the above embodiments.

[0162] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0163] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0164] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer storage media, and computer program products are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0165] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A multimodal device, characterized in that, The multimodal device includes: The central management module is used to receive mode selection signals and dynamically switch the logical connection relationship of physical storage sub-units between a first working mode and a second working mode according to the mode selection signals. The first working mode and the second working mode have different physical storage resource requirements. The mode selection signals are used to indicate whether to work in the first working mode or in the second working mode. Each physical storage sub-unit is configured to be invoked in both the first working mode and the second working mode. A physical storage unit includes multiple physical storage sub-units. At least one physical storage sub-unit is time-division multiplexed to serve the first working mode or the second working mode according to the logical connection relationship. In this case, two or more physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal by bit-width splicing and / or depth splicing.

2. The multimode device according to claim 1, characterized in that, The central management module selects two or more physical storage sub-units for logical concatenation to meet the physical storage resource requirements under the current working mode; The logical concatenation includes at least one of the following methods: Bit-width splicing combines at least two physical storage sub-units in the bit-width dimension, so that the spliced ​​bit-width is not less than the bit-width requirement of the current working mode; Depth splicing combines at least two physical storage sub-units in the depth dimension, ensuring that the depth of the spliced ​​unit is not less than the depth requirement of the current working mode. Hybrid splicing combines at least two physical storage sub-units simultaneously in both the bit width and depth dimensions, ensuring that the bit width and depth of the spliced ​​unit are not lower than the corresponding requirements of the current working mode.

3. The multimode device according to claim 1, characterized in that, The bit width and / or depth specifications of some of the physical storage sub-units are configured to be a common multiple of the maximum storage unit requirements of the first operating mode and the second operating mode, so that a single physical storage sub-unit can meet the requirements of the two modes respectively through time-division multiplexing.

4. The multimode device according to claim 1, characterized in that, The bit width and / or depth specifications of some of the physical storage sub-cells are configured to serve the current operating mode, either alone or in combination with other physical storage sub-cells.

5. The multimode device according to claim 1, characterized in that, The multimodal device also includes: The first modal core is used to operate in the first working mode based on the physical storage sub-units connected by the logical connection relationship; The second modal core is used to operate in the second working mode based on the physical storage sub-units connected by the logical connection relationship.

6. The multimode device according to claim 1, characterized in that, The multi-mode device has a preset address mapping logic for physical storage sub-units, which is used to directly activate the corresponding physical storage sub-unit combination path according to the mode selection signal during the operation phase. Through bit-width splicing and / or depth splicing, multiple physical storage sub-units are combined into a storage unit that meets the requirements of the current working mode.

7. A method for accessing storage resources of a multimodal device, characterized in that, The method includes: A receiving mode selection signal is received, the mode selection signal being used to indicate whether to operate in a first operating mode or in a second operating mode; The logical connection relationship of physical storage sub-units is dynamically switched between the first working mode and the second working mode according to the mode selection signal. The first working mode and the second working mode have different physical storage resource requirements. Each physical storage sub-unit is configured to be invoked in both the first working mode and the second working mode. According to the logical connection relationship, at least one physical storage sub-unit in the physical storage unit is called to realize the time-division multiplexing of the physical storage sub-unit to serve the first working mode or the second working mode. In this case, by bit-width splicing and / or depth splicing, two or more called physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal.

8. The storage resource retrieval method according to claim 7, characterized in that, The logical connection relationship of dynamically switching physical storage sub-units between the first operating mode and the second operating mode according to the mode selection signal includes: Select two or more of the aforementioned physical storage sub-units for logical concatenation to meet the physical storage resource requirements under the current working mode; The logical concatenation includes at least one of the following methods: Bit-width splicing combines at least two physical storage sub-units in the bit-width dimension, so that the spliced ​​bit-width is not less than the bit-width requirement of the current working mode; Depth splicing combines at least two physical storage sub-units in the depth dimension, ensuring that the depth of the spliced ​​unit is not less than the depth requirement of the current working mode. Hybrid splicing combines at least two physical storage sub-units simultaneously in both the bit width and depth dimensions, ensuring that the bit width and depth of the spliced ​​unit are not lower than the corresponding requirements of the current working mode.

9. The storage resource retrieval method according to claim 7, characterized in that, The logical connection relationship of dynamically switching physical storage sub-units between the first operating mode and the second operating mode according to the mode selection signal includes: In both the first and second working modes, the same physical storage sub-unit is called in a time-division multiplexing manner to meet the storage requirements of that mode; The bit width and / or depth of the physical storage sub-unit are common multiples of the maximum storage unit requirements of the first operating mode and the second operating mode.

10. The storage resource retrieval method according to claim 7, characterized in that, Calling at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship includes: A single physical storage subunit can be used to meet the storage requirements of this mode; or This physical storage subunit is combined with other physical storage subunits for use to meet the storage requirements of this mode.

11. The storage resource retrieval method according to claim 7, characterized in that, Calling at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship includes: During the operation phase, according to the mode selection signal, a preset address mapping logic is invoked, which includes the physical storage sub-unit to be invoked and its address. Based on the address mapping logic, the corresponding physical storage sub-unit combination path is directly activated; By splicing bit width and / or depth, multiple physical storage sub-units are dynamically combined into storage units that meet the requirements of the current working mode.

12. A method for configuring storage resources in a multimodal device, characterized in that, The method includes: Obtain the physical storage resource requirements of the first working mode and the second working mode respectively, the requirements including bit width requirements and depth requirements; According to the requirements specification, the physical storage unit is divided into multiple physical storage sub-units, and the bit width specification and / or depth specification of each physical storage sub-unit are configured. Each physical storage sub-unit is configured to be invoked in the first working mode and the second working mode. Establish logical connections between the physical storage sub-unit and the first working mode and the second working mode, so that at least one physical storage sub-unit can be time-division multiplexed under different working modes. In this way, two or more physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal by bit width splicing and / or depth splicing.

13. The storage resource configuration method according to claim 12, characterized in that, Establishing logical connections includes: Configure logical splicing relationships between two or more physical storage sub-units to meet the physical storage resource requirements of the current working mode; The logical concatenation relationship includes at least one of the following methods: Configure the bit width splicing relationship to combine at least two physical storage sub-units in the bit width dimension, so that the bit width of the spliced ​​unit is not lower than the bit width requirement of the current working mode; Configure depth-based splicing relationships to combine at least two physical storage sub-units in the depth dimension, ensuring that the depth of the spliced ​​unit is not less than the depth requirement of the current working mode; Configure a hybrid splicing relationship, combining at least two physical storage sub-units simultaneously in both bit width and depth dimensions, so that the bit width and depth of the spliced ​​unit are not lower than the corresponding requirements of the current working mode.

14. The storage resource configuration method according to claim 12, characterized in that, The configuration of the bit width and / or depth specifications of each physical storage sub-unit includes: Analyze the maximum image width supported by the first working mode and the second working mode respectively, and find the common multiple relationship between the maximum image widths; When a common multiple relationship exists, the bit width and / or depth specifications of a physical storage sub-unit are configured to be the common multiple, so that the physical storage sub-unit can be used in both operating modes.

15. The storage resource configuration method according to claim 12, characterized in that, Establishing the logical connection relationship between the physical storage sub-unit and the first working mode and the second working mode includes: Generate a first storage requirement list for the first working mode and a second storage requirement list for the second working mode; Greedily match the unmatched storage requirement items in the first storage requirement list with the storage requirement items in the second storage requirement list; The greedy matching includes at least one of the following methods: One-to-one matching, ensuring that each item in the first storage requirement list meets the depth and bit width requirements of the second storage requirement list; Many-to-one splicing and matching involves splicing and combining multiple storage requirement items in the second storage requirement list in the bit width dimension and / or depth dimension, and pairing them with one item in the first storage requirement list. One-to-many splitting and matching involves splitting a large storage requirement in the first storage requirement list into multiple sub-requirements, which are then paired with multiple storage requirement items in the second storage requirement list.

16. The storage resource configuration method according to claim 15, characterized in that, Before the greedy matching is executed, the following steps are also included: Sort the first storage requirement list from largest to smallest by depth and / or bit width; Sort the second storage requirement list in ascending order of depth and / or bit width; Specifically, the first storage requirement list after being sorted by one-to-one matching is used to find the smallest storage requirement item in the second storage requirement list after being sorted, whose depth and bit width are both greater than or equal to the current item, and then pair them.

17. The storage resource configuration method according to claim 12, characterized in that, After establishing the logical connections, the following is also included: The logical connection relationship is solidified into a preset address mapping logic, so that during the operation phase, the corresponding physical storage sub-unit combination path is directly activated according to the mode selection signal. Through bit width splicing and / or depth splicing, multiple physical storage sub-units are dynamically combined into storage units that meet the requirements of the current working mode.

18. A multimodal storage resource retrieval device, characterized in that, The device includes: The central management module is used to receive a mode selection signal, which indicates whether to operate in a first operating mode or a second operating mode. The storage switching module is used to dynamically switch the logical connection relationship of physical storage sub-units between the first working mode and the second working mode according to the mode selection signal, wherein the first working mode and the second working mode have different physical storage resource requirements, and each physical storage sub-unit is configured to be invoked in both the first working mode and the second working mode; The storage multiplexing module calls at least one physical storage sub-unit in the physical storage unit according to the logical connection relationship, so as to realize the time-division multiplexing of the physical storage sub-unit to serve the first working mode or the second working mode. In this way, by bit width splicing and / or depth splicing, the two or more called physical storage sub-units are combined into a storage unit that meets the working mode requirements selected by the mode selection signal.

19. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 7-17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 7-17.

Images