A vision-sensing array-based light-to-word conversion system and method

Abstract

The application discloses a kind of light to word conversion system and method based on visual sensing array, including visual sensing array, gating unit, input circuit, readout circuit and control circuit, visual sensing array is composed of multiple visual sensing units, each visual sensing unit is used to obtain the visual information of its spatial position, and form the physical state representing the visual information obtained, gating unit is used to configure the image block in visual sensing array according to the word generation mode of artificial intelligence model, and the visual sensing unit that forms target image block is gated, input circuit is used to load the input signal obtained by embedding parameter conversion determined in advance by artificial intelligence model to target image block, readout circuit is used to summarize the output response of multiple visual sensing units in target image block, and generate word embedding component.The present application realizes the direct conversion of optical signal to word in the process of visual information acquisition, and provides word input for artificial intelligence model.

Description

Technical Field

[0001] This invention relates to the field of visual sensing technology, and in particular to a light-to-word conversion system and method based on a visual sensing array. Background Technology

[0002] Visual information is a crucial source of information about the external environment for artificial intelligence systems. Visual sensors are responsible for converting external light signals into image data, providing visual input for subsequent AI models. With the development of AI technology, words have become the basic data units widely used in modern AI models; visual information needs to be converted into words before it can be input into AI models for tasks such as understanding, reasoning, and decision-making.

[0003] Existing technologies typically involve first acquiring images using a visual sensor, then converting the image data into tokens using a digital processor, and finally inputting these tokens into an artificial intelligence model. This process involves multiple stages, including image acquisition, data readout, data transmission, and digital processing. The separation of visual information acquisition and token generation leads to frequent transmission of visual information between different hardware modules, increasing data movement and computational overhead, and making it difficult to meet the demands of intelligent vision systems for efficient and low-latency visual information processing.

[0004] Therefore, a technology is needed that can directly convert light into words during the visual information acquisition process. Summary of the Invention

[0005] Purpose of the invention: The first purpose of this invention is to provide a light-to-word conversion system and method based on a visual sensing array, so as to realize the direct conversion of light signals to words during the acquisition of visual information.

[0006] Technical solution: To achieve the above objectives, the present invention provides a light-to-word conversion system based on a visual sensing array, comprising a visual sensing array, a gating unit, an input circuit, and a readout circuit;

[0007] The visual sensing array includes multiple visual sensing units for acquiring visual information about the spatial location.

[0008] The gating unit is connected to the visual sensing array and selects the visual sensing units that make up the target image block; the image block is configured in the visual sensing array according to the word generation method of the artificial intelligence model, and the target image block refers to the image block selected by the gating unit for calculation.

[0009] The input circuit is connected to the visual sensing array and loads an input signal onto the target image block. The input signal is obtained by converting the embedding parameters predetermined by the artificial intelligence model.

[0010] The readout circuit is connected to the output terminal of the visual sensing array, and summarizes the output response of the visual sensing array to generate word embedding components.

[0011] Preferably, it also includes a control circuit for coordinating the operating timing between the visual sensor array, the gating unit, the input circuit, and the readout circuit.

[0012] Preferably, the visual sensing unit is a photosensitive storage device, which includes a phototransistor, a photosensitive memristor, or a photosensitive ferroelectric device.

[0013] Preferably, the visual sensing unit is composed of a cascaded photosensitive device and a memory device, wherein the photosensitive device includes a photodiode or a phototransistor, and the memory device includes a floating gate memory, a memristor, a resistive random access memory, or a ferroelectric memory.

[0014] Preferably, the visual sensing array is a photosensitive storage array composed of floating gate phototransistors.

[0015] Preferably, the plurality of visual sensing units form a physical state characterizing the acquired visual information, the physical state including charge quantity, conductance, threshold voltage or current response.

[0016] Preferably, the embedded parameters are converted into input signals for the visual sensing array based on the operating range of the visual sensing array, and the input signals include voltage, current, pulse, charge, frequency, or time-coded signals.

[0017] Preferably, the output response of the visual sensing unit is generated as a lexical embedding component using Kirchhoff current summation.

[0018] The present invention discloses a light-to-lexical conversion method based on a visual sensor array, which generates complete lexical units by sequentially or in parallel inputting multiple sets of embedding parameters, including:

[0019] S1. Light information from the external environment is projected onto the visual sensing array, changing the physical state of each visual sensing unit.

[0020] S2. Based on the word generation method of the artificial intelligence model, configure image blocks in the visual sensor array;

[0021] S3. Use the gating unit to select the visual sensing units that make up the target image block;

[0022] S4. The input signal is loaded onto the target image block using the input circuit, and the visual sensing unit generates an output response based on its own physical state and the input signal.

[0023] S5. The outputs of multiple visual sensing units are summed using the readout circuit to generate embedded components of word units.

[0024] Repeat steps S2 to S5 for different image blocks in the visual sensor array to obtain the word sequence of the input visual information.

[0025] Preferably, visual sensing units with the same relative position in multiple target image blocks share the same embedding parameter input to perform parallel computation on the same embedding dimension for multiple image blocks.

[0026] Beneficial effects: The present invention has the following advantages: The present invention simplifies the multi-level processing flow of image acquisition, digital processing and word generation in traditional vision systems, and can realize the direct conversion of light signals to words during the acquisition of visual information, providing word input for artificial intelligence models, realizing efficient connection between visual sensor output and artificial intelligence model input, reducing data transfer and overall system energy consumption, improving data interaction efficiency, and also reducing the hardware complexity required to realize visual information word conversion, which is conducive to building a more compact intelligent vision hardware system. Attached Figure Description

[0027] Figure 1 This is a schematic diagram illustrating the principle of light-to-word conversion based on a visual sensor array.

[0028] Figure 2 This is a schematic diagram of a light-to-word conversion system architecture based on a visual sensor array;

[0029] Figure 3 This is a schematic diagram of the light-to-word conversion workflow based on a visual sensor array;

[0030] Figure 4 A schematic diagram of key performance parameters for a visual sensor array;

[0031] Figure 5 This is a schematic diagram of the output results of the light-to-word conversion;

[0032] Figure 6 A schematic diagram illustrating the results of using word segments output by a visual sensor array for inference by an artificial intelligence model.

[0033] Figure 7 This is a schematic diagram of the circuit structure for realizing light-to-word conversion based on a photosensitive storage array;

[0034] Figure 8 This is a schematic diagram of the fabrication process for a photosensitive memory array. Detailed Implementation

[0035] The technical solution of the present invention will be described in detail below with reference to the embodiments and accompanying drawings.

[0036] Example 1

[0037] like Figure 1 , 2As shown, this embodiment provides a light-to-word conversion system based on a visual sensor array. The light-to-word conversion refers to the visual sensor array directly generating multi-dimensional words corresponding to each image block after receiving an incident light signal, without first transmitting the complete image data to a separate digital processor for word generation. The system includes a visual sensor array and a gating unit, input circuit, readout circuit, and control circuit connected to the visual sensor array. The image block refers to a basic word processing unit composed of visual sensor units arranged in a regular pattern; one image block generates one word sequence. The target image block refers to the image block selected by the gating unit for calculation.

[0038] The visual sensing array includes multiple visual sensing units (pixels) arranged in spatial order. Each visual sensing unit is used to sense the incident light signal at its corresponding spatial position and form a physical state corresponding to the local visual information. The physical state can be expressed as charge, conductance, threshold voltage, current response, or other device states that can characterize the incident light information.

[0039] The visual sensing unit can be a photosensitive memory device, or it can be composed of a cascaded photosensitive device and a memory device. The photosensitive memory device includes a phototransistor, a photomemristor, a photoferroelectric device, or other photosensitive memory devices with photoresponse and state retention capabilities; the photosensitive device includes a photodiode, a phototransistor, or other photoelectric conversion device; the memory device includes a floating-gate memory, a memristor, a resistive random access memory, a ferroelectric memory, or other memory devices with state retention capabilities.

[0040] In a preferred embodiment, the visual sensing array employs a photosensitive storage array composed of floating-gate phototransistors. The floating-gate phototransistors utilize two-dimensional semiconductor channels to acquire visual information and employ a floating-gate structure to form non-volatile physical states. The photosensitive storage array achieves embedded parameter loading and parallel lexical generation through multi-layer metal interconnects. The generated lexicals can be directly input into a visual Transformer, a large visual model, or other data processing models that use lexicals as input, realizing an integrated hardware system for visual information acquisition, lexical generation, and artificial intelligence inference.

[0041] The gating unit is used to select multiple visual sensing units in the target image block and apply or apply in parallel input signals corresponding to the embedding parameters to the visual sensing units corresponding to the target image block. The embedding parameters are predetermined by the artificial intelligence model and mapped to the input signals of the visual sensing array according to the working range of the visual sensing array; the input signals mapped by the embedding parameters include voltage, current, pulse, charge, frequency, time-coded signals, or other information carriers that can characterize the embedding parameters.

[0042] The input circuit is used to load the input signal, which is converted by the embedding parameters predetermined by the artificial intelligence model, onto the target image block.

[0043] The readout circuit is used to summarize the output responses of multiple visual sensing units within the target image block and generate word embedding components. Specifically, word embedding components can be generated using an analog summarization method, preferably using Kirchhoff current summarization.

[0044] The control circuit is used to coordinate the timing of operation between the vision sensor array, gating unit, input circuit and readout circuit.

[0045] In this embodiment, the visual sensing units in the selected image block generate corresponding outputs based on their light response state and the applied input signal. The outputs of multiple visual sensing units are aggregated to form an embedding component of a word, which corresponds to one dimension of the image block in the embedding space. By applying multiple sets of embedding parameters sequentially or in parallel, multiple embedding components corresponding to the target image block can be obtained, and these multiple embedding components constitute a word.

[0046] Repeating the above operations on different image blocks in the visual sensing array yields a word sequence corresponding to the input visual information. In specific implementations, the image block gating, embedding parameter input, operation timing control, and readout functions can be implemented by addressing circuits, input circuits, control circuits, and readout circuits, respectively, or they can be jointly implemented by corresponding functional units in an integrated circuit. The aforementioned supporting circuits are used to complete the gating and operation of the visual sensing array, and their specific circuit configuration does not constitute a limitation on the light-to-word conversion system.

[0047] In a visual artificial intelligence model that uses lexical units as input, the input image can be divided into multiple image blocks. Each image block, after undergoing the same embedding mapping, forms a multidimensional vector, which constitutes the lexical unit corresponding to the image block. The lexical units corresponding to multiple image blocks are arranged in a predetermined order to form a lexical sequence input to the artificial intelligence model. The size, position, number, and arrangement of the image blocks can be configured according to the lexical generation requirements of the artificial intelligence model.

[0048] The system described in this embodiment receives light signals from the physical world and directly converts these signals into tokens that can be processed by an artificial intelligence model, thereby establishing a visual information interface between the physical world and the artificial intelligence model. Unlike the traditional processing method where a visual sensor first generates complete image data and then a digital processor generates tokens, this embodiment completes visual information acquisition and token generation at the visual sensing end, transforming the output of the visual sensing system from image data into tokens required by the artificial intelligence model.

[0049] Example 2

[0050] like Figure 3 As shown, this embodiment, based on the system described in Embodiment 1, provides a light-to-lexical conversion method, including steps such as visual information acquisition, visual information state formation, image block configuration and gating, embedding parameter input, lexical generation, and lexical sequence output. Figure 3 As shown in a, the specific steps include the following:

[0051] S1. Visual information acquisition.

[0052] A visual sensing array acquires external visual information, and each visual sensing unit receives visual signals corresponding to its spatial location. This visual information can be light intensity information, polarization information, spectral information, depth information, or other visual information that can be acquired by the visual sensing array. This embodiment uses light intensity information as the visual information for explanation.

[0053] S2, Visual information state formation.

[0054] Each visual sensing unit forms a physical state corresponding to the local visual information based on the received visual signal. The physical state can be a charge quantity, conductance, threshold voltage, current response, or other device states that can characterize the visual information. Preferably, the physical state is directly stored inside the visual sensing unit, and the acquisition and storage of visual information are completed in the same device, reducing data readout and transmission processes and improving system processing efficiency. In other embodiments, a combination of a photosensitive sensor and a memory can also be used to achieve the formation and maintenance of the visual information state; as long as the subsequent word generation process can be completed, it falls within the protection scope of this invention.

[0055] S3, Image Block Configuration and Gating.

[0056] Based on the requirements of the artificial intelligence model for word generation, image blocks are configured on the visual sensing array, and multiple visual sensing units corresponding to the target image blocks are selected. The size, position, number, and arrangement of the image blocks can all be configured according to different artificial intelligence models. Image blocks can be non-overlapping or partially overlapping; they can be divided in a fixed way or dynamically configured according to application requirements. This invention does not limit the configuration method of image blocks, as long as it can generate words corresponding to the input requirements of the artificial intelligence model.

[0057] S4, Embedded parameter input.

[0058] An input signal corresponding to the embedding parameters is applied to the visual sensing unit corresponding to the target image block, causing each visual sensing unit to generate a corresponding output based on its physical state and the input signal. The embedding parameters can be predetermined by an artificial intelligence model and converted into corresponding input signals according to the device characteristics of the visual sensing array. Preferably, the embedding parameters are electrically mapped according to the operating voltage range, response characteristics, and dynamic range of the visual sensing array, so that the mapped input signal matches the operating range of the visual sensing array. The input signal can be voltage, current, pulse, or other signal forms capable of embedding mapping; its specific implementation does not constitute a limitation of the present invention. Preferably, visual sensing units at the same location can simultaneously receive the same embedding parameter input, enabling multiple image blocks to complete the calculation of the same embedding dimension in parallel, thereby improving the efficiency of word generation. Figure 3 b illustrates a representative example of embedding parameter input, where a 3×3×d embedding parameter matrix is ​​converted into a 9×d voltage sequence, which is simultaneously input into n image patches for parallel computation.

[0059] S5, Lexical generation.

[0060] Each visual sensing unit within the target image block is input with its corresponding embedding parameters. Each visual sensing unit generates an output response based on its stored physical state and the corresponding input signal. The output responses of multiple visual sensing units within the target image block are then summed to obtain an embedding component of a word. Preferably, the output responses are summed using Kirchhoff's current law to achieve a simulated summation of the outputs from multiple visual sensing units within the target image block, such as... Figure 3 As shown in c; in other embodiments, voltage summation, charge integration, digital accumulation, or other methods capable of output summarization can also be used to generate embedded components, and this invention is not limited to this. By inputting multiple sets of embedding parameters sequentially or in parallel, multiple embedded components can be obtained, and the multiple embedded components constitute the lexical units corresponding to the target image block.

[0061] S6, output the word sequence.

[0062] like Figure 3 As shown in d, by adjusting the gating state of the visual sensing array, other image blocks are selected sequentially, and the above steps are repeated to obtain the corresponding word units for each image block. The word units corresponding to the multiple image blocks are output in a predetermined order to form a word unit sequence corresponding to the input visual information, and then output to the subsequent artificial intelligence model for processing. Preferably, the word unit sequence is output according to the spatial arrangement order of the image blocks; in other embodiments, it can also be configured according to the requirements of different artificial intelligence models for word unit arrangement, and this invention does not limit this. The output of the word unit sequence has temporal characteristics, and its output duration is jointly determined by the parallel computing power of the visual sensing array, the number of image blocks, and the scale of the embedding parameters.

[0063] like Figure 4 As shown, to achieve lexical generation in the light-to-lexical conversion, the visual sensing array should possess corresponding functional characteristics. For example... Figure 4 As shown in c, visual information is first programmed, and then the embedded parameters are input to enter the lexical calculation stage.

[0064] First, the visual sensing array should be able to form a physical state corresponding to the acquired visual information and maintain this physical state during subsequent lexical generation. For example... Figure 4 As shown in a-1, visual information is stored in each visual sensing unit in the form of charge states; further, as Figure 4 As shown in b, under different light intensity programming conditions, the device readout response maintains a good linear relationship with the light intensity and has positive and negative symmetry, ensuring that different visual information can be stably mapped to the corresponding device state.

[0065] Secondly, the visual sensing unit should be able to respond to the embedded parameter input, so that the device output is simultaneously modulated by the physical state corresponding to the visual information and the embedded parameters, thereby realizing the embedded mapping of visual information to the word space. Figure 4 Taking the output current shown in a-2 as an example, by inputting the embedding voltage, in-situ embedding calculations can be performed on multiple programmed visual sensing units within an image block. The output current can be expressed as the calculation result of the combined effect of visual information and embedding parameters. Furthermore, as shown in a-2... Figure 4 As shown in d, under the same visual information programming conditions, the device output response maintains a good linear relationship and positive / negative symmetry with the input parameters, thereby ensuring accurate generation of lexical embedding components. Preferably, the output response has continuously adjustable analog characteristics to improve the accuracy and representational capability of lexical generation.

[0066] In addition to the above functional requirements, the visual sensing array is preferably characterized by good array consistency, response stability, and device durability uniformity, so that different visual sensing units can produce consistent output responses under the same input conditions, thereby improving the stability of word generation results.

[0067] like Figure 5 As shown, this embodiment completes the direct conversion of visual information into word units. Figure 5 a demonstrates the input visual information stored in the visual sensor array. Based on this visual information, it can be divided into multiple 3x3 image blocks according to a preset method, such as... Figure 5 As shown in b. The same embedding matrix is ​​input for different image patches ( Figure 5 c) After that, the visual sensor array outputs the corresponding word units, such as Figure 5As shown in d, since different image patches contain different visual information, their output lexical units also have different response characteristics, thus forming a lexical sequence corresponding to the input visual information. The above results show that the present invention does not require outputting a complete image and relying on an independent digital processor to complete lexical generation, but directly completes the light-to-lexical conversion within the visual sensing array, providing directly usable lexical input for artificial intelligence models.

[0068] Example 3

[0069] This embodiment uses the lexical units output by the visual sensor array directly as input to the artificial intelligence model to verify the effectiveness and usability of the lexical units generated in Embodiment 2. Figure 6 As shown.

[0070] The artificial intelligence model is a data processing model that takes lexical units as input. Lexical units output by the visual sensor array can be directly input into the artificial intelligence model, or they can be input into the model after data format conversion, normalization, or quantization according to the model's input requirements.

[0071] In this embodiment, the Vision Transformer (ViT) model is used to perform image classification verification on the lexical units output in Example 2. Figure 6 a). Furthermore, in this embodiment, the optical signal is established using the visual sensing array device response obtained from experimental measurements, the mapping relationship between the embedded parameters and the output response is established, and the hardware parameter matching between the visual sensing array and the artificial intelligence model is completed accordingly. Figure 6 b). After the visual sensing array completes the light-to-lexical conversion, the generated lexical sequence is directly input into the model for inference to obtain the image classification result. Figure 6 ce).

[0072] Experimental results show that the lexical units generated in Example 2 can effectively preserve the features of the input visual information, and their image classification accuracy is basically consistent with that of the pure software lexical unit generation scheme. Figure 6 f) indicates that the generated lexical units can be directly used as effective input to the artificial intelligence model. Simultaneously, since the lexical units are generated directly within the visual sensing array, the traditional image readout, data handling, and digital lexical unit generation processes are reduced. While maintaining the model's recognition performance essentially unchanged, the overall energy consumption of the evaluation system is reduced by approximately 14.3 times, verifying that this invention can significantly improve the energy efficiency of the intelligent vision system while achieving direct light-to-lexical unit conversion. Figure 6 g).

[0073] Example 4

[0074] This embodiment uses an optical storage array as an example to implement the light-to-word conversion system proposed in Embodiment 1, such as... Figure 7As shown, the visual sensing array consists of multiple photosensitive storage units arranged in a two-dimensional array. Each visual sensing unit includes a floating-gate phototransistor with photoresponse and non-volatile storage capabilities. Each visual sensing unit is connected to a row selection signal, a column selection signal, an embedded parameter input terminal, and an output readout terminal, respectively. The image block gating, embedded parameter input, output readout, and operation timing control are completed through external control circuitry.

[0075] The peripheral control circuit includes an addressing circuit, an input circuit, a control circuit, and a readout circuit. The addressing circuit is used to control the gating of the visual sensor array according to the image block configuration; the input circuit is used to load the input signal corresponding to the embedding parameters into the target image block; the control circuit is used to control the timing of operations such as visual information programming, word calculation, and readout; and the readout circuit is used to summarize the output responses of each visual sensor unit in the target image block and output the corresponding word embedding components.

[0076] Preferably, visual sensing units with the same relative position in multiple image blocks can simultaneously receive the same embedding parameter input, enabling parallel computation of multiple image blocks corresponding to the same embedding dimension and improving word generation efficiency. Switching between image blocks is accomplished through an external gating circuit, with different image blocks sequentially completing word generation to form a complete word sequence.

[0077] Example 5

[0078] This embodiment provides a method for fabricating the photosensitive storage array described in Embodiment 4. For example... Figure 8 As shown, this embodiment fabricates a photosensitive memory array composed of a single-layer molybdenum sulfide floating gate phototransistor. The single-layer molybdenum sulfide serves as the photosensitive channel, exhibiting good photoelectric response characteristics and contributing to the uniformity of the array-level device. The device employs a molybdenum sulfide / alumina / gold nanoparticle / alumina heterostructure, where the gold nanoparticle layer acts as the floating gate for storing photogenerated charges. The fabrication of the photosensitive memory array includes the following steps:

[0079] (1) Substrate preparation and floating gate structure fabrication.

[0080] A p-type silicon substrate with a 280 nm silicon dioxide layer was selected. The pattern of the first metal layer M1 was defined by photolithography, and 1 nm of titanium and 25 nm of gold were deposited sequentially by electron beam evaporation. After lift-off, the first metal layer was formed. Subsequently, the substrate was plasma-cleaned, a 1 nm aluminum seed layer was deposited, and a 22 nm alumina dielectric layer was formed by atomic layer deposition, with the first 8 nm deposited at 120 °C and the remaining 14 nm deposited at 250 °C. The same photolithography, metal evaporation, aluminum seed layer deposition, and alumina atomic layer deposition process was used to form the second metal layer M2 and its dielectric layer, which consisted of 1 nm of titanium and 30 nm of gold. Subsequently, a 1.5 nm thick gold nanoparticle floating gate layer was formed by electron beam evaporation, and an 8 nm alumina tunneling dielectric layer was formed by atomic layer deposition at 150 °C.

[0081] (2) Formation of through holes.

[0082] The via region connecting the first metal layer M1 and the third metal layer M3 is defined by photolithography, and the dielectric material in the via region is removed using an inductively coupled plasma etching process with carbon tetrafluoride and oxygen as the working gas. After etching, the substrate is placed in hot acetone at 55 °C and subjected to ultrasonic treatment to remove residual photoresist. The via is used to achieve the electrical connection between M1 and M3 in the subsequent fabrication of the third metal layer.

[0083] (3) Molybdenum sulfide transfer and active region formation.

[0084] A polymethyl methacrylate (PMMA)-assisted wet transfer process was used to transfer a monolayer molybdenum sulfide (MoS) film onto a target substrate with a pre-formed floating gate structure and vias. Specifically, PMMA was spin-coated onto the MoS film surface, and the film was then placed in deionized water to separate the PMMA / MoS stack from the native substrate. The stack was then transferred to the target substrate and heated at 120 °C for 20 min under vacuum to improve interface adhesion and remove moisture. The sample was then immersed in hot acetone at 80 °C for 30 min to remove the PMMA. MoS channel regions were defined using electron beam lithography, and the MoS outside the channel regions was removed using oxygen reactive ion etching at 10 mTorr pressure and 100 W power, thus forming the active channels for each visual sensing unit.

[0085] (4) Formation of source and drain electrodes and third metal layer.

[0086] The source / drain electrodes and the third metal layer M3 were patterned using electron beam lithography. Subsequently, 30 nm antimony and 40 nm gold were deposited sequentially using thermal evaporation, and source / drain contacts and array interconnects were formed through a lift-off process. The total thickness of the third metal layer is greater than the via depth to ensure a reliable electrical connection between M1 and M3. Interdigitated electrodes are also incorporated into the third metal layer to increase the effective photosensitive area of ​​the visual sensing unit.

[0087] In this embodiment, M1, M2, and M3 together constitute a multi-layer interconnect structure of the array, providing electrical connections for independent programming of each visual sensing unit, image patch gating, embedding parameter loading, and output readout. The specific interconnect relationships and their roles in parallel image patch embedding are discussed in detail below. Figure 7 The circuit structure shown is explained.

[0088] In some implementations, the peripheral control circuitry can be implemented using application-specific integrated circuits (ASICs) and monolithically or three-dimensionally integrated with the visual sensing array. Preferably, the addressing circuit, input circuit, control circuit, and readout circuitry can be integrated into a silicon-based circuit layer below the visual sensing array and connected to the upper-layer visual sensing array via a vertical interconnect structure to shorten the signal transmission path and improve system integration and data transmission efficiency. Furthermore, the light-to-word conversion system can serve as the front-end visual interface of an artificial intelligence hardware system, and can be integrated on-chip or in a package-level with the back-end artificial intelligence model computing unit. The artificial intelligence model computing unit can employ a visual Transformer, a large visual model, or other data processing models that use words as input to realize an integrated hardware system for visual information acquisition, word generation, and artificial intelligence inference, thereby further reducing data handling overhead and improving overall system energy efficiency.

Claims

1. A light-to-word conversion system based on a visual sensor array, characterized in that, Includes a visual sensor array, gating unit, input circuit, and readout circuit; The visual sensing array includes multiple visual sensing units for acquiring visual information about the spatial location. The gating unit is connected to the visual sensing array and gating the visual sensing units that make up the target image block. The image block is configured in the visual sensing array according to the word generation method of the artificial intelligence model. The target image block refers to the image block selected by the gating unit for calculation. The input circuit is connected to the visual sensing array and loads an input signal onto the target image block. The input signal is obtained by converting the embedding parameters predetermined by the artificial intelligence model. The readout circuit is connected to the output terminal of the visual sensing array, and summarizes the output response of the visual sensing array to generate word embedding components.

2. The optical-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, It also includes control circuitry for coordinating the timing of the operation of the vision sensor array, gating unit, input circuitry, and readout circuitry.

3. The optical-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, The visual sensing unit is a photosensitive storage device, which includes a phototransistor, a photosensitive memristor, or a photosensitive ferroelectric device.

4. The light-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, The visual sensing array is a photosensitive storage array composed of floating gate phototransistors.

5. The optical-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, The visual sensing unit is composed of a cascaded photosensitive device and a memory device. The photosensitive device includes a photodiode or a phototransistor, and the memory device includes a floating gate memory, a memristor, a resistive random access memory, or a ferroelectric memory.

6. The optical-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, The plurality of visual sensing units form a physical state that characterizes the acquired visual information, the physical state including charge quantity, conductance, threshold voltage or current response.

7. The optical-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, The embedded parameters are converted into input signals for the visual sensing array based on the operating range of the visual sensing array. The input signals include voltage, current, pulse, charge, frequency, or time-coded signals.

8. The light-to-word conversion system based on a visual sensor array according to claim 1, characterized in that, The output response of the visual sensing unit is generated as a lexical embedding component using Kirchhoff current summarization.

9. A conversion method for a light-to-word conversion system based on a visual sensor array as described in claim 1, characterized in that, Complete lexical units are generated by sequentially or in parallel inputting multiple sets of embedding parameters, including: S1. Light information from the external environment is projected onto the visual sensing array, changing the physical state of each visual sensing unit. S2. Based on the word generation method of the artificial intelligence model, configure image blocks in the visual sensor array; S3. Use the gating unit to select the visual sensing units that make up the target image block; S4. The input signal is loaded onto the target image block using the input circuit, and the visual sensing unit generates an output response based on its own physical state and the input signal. S5. The outputs of multiple visual sensing units are summed using the readout circuit to generate embedded components of word units. Repeat steps S2 to S5 for different image blocks in the visual sensor array to obtain the word sequence of the input visual information.

10. The light-to-word conversion method based on a visual sensor array according to claim 9, characterized in that, Visual sensing units with the same relative position in multiple target image patches share the same embedding parameter input to perform parallel computation on the same embedding dimension for multiple target image patches.

Images