Visual image monitoring method and electronic chip

Abstract

The invention relates to the technical field of image processing. The invention provides a visual image monitoring method and an electronic chip, and the method comprises the steps: collecting an original image flow through a complementary metal oxide semiconductor sensor of a bionic retina structure, and generating a dynamic perception event flow; carrying out target detection and trajectory prediction processing, and generating a vector data set comprising a space-time correlation mark; performing heterogeneous graphic rendering processing on the original image flow through a programmable rendering pipeline to generate a superimposed image fused with augmented reality annotations; carrying out self-adaptive transparency mixing processing on the superposed image, the middle feature layer and the environment state layer to generate a feedback control signal and a visual synthetic image; and synchronous output processing is carried out on the visual synthetic image and the feedback control signal through the photon coding interface to obtain a synchronous output signal, so that the problem that dynamic transparency mixing and photon-level low-delay output cannot be supported in related technologies is solved, and the requirements of high-precision real-time monitoring and closed-loop control in an industrial scene are met.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method for monitoring visual images and an electronic chip. Background Art

[0002] In the fields of industrial inspection and automation control, visual image monitoring technology, as a core perception means, plays a decisive role in real-time performance, accuracy, and system efficiency. In the traditional visual image monitoring technology system, although there are already monitoring methods and hardware solutions based on machine vision, this technology has significant limitations. For example, in some high-end scenarios (such as semiconductor defect detection and high-speed robotic arm control), it is necessary to achieve the collaborative optimization of high-frame-rate image acquisition, multi-dimensional feature fusion, and dynamic resource scheduling, and it is expected to synchronously process multiple tasks such as target detection, trajectory prediction, and enhanced visualization during the monitoring process, and dynamically adjust the algorithm priority and hardware resource allocation according to the real-time environmental state.

[0003] However, the traditional method cannot support dynamic transparency blending and photon-level low-latency output, making it difficult to meet the requirements of high-precision real-time monitoring and closed-loop control in industrial scenarios. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method for monitoring visual images and an electronic chip to solve the problem in the related technology that dynamic transparency blending and photon-level low-latency output cannot be supported, and to meet the requirements of high-precision real-time monitoring and closed-loop control in industrial scenarios.

[0005] In a first aspect, the present application provides a method for monitoring visual images, including:

[0006] Collecting an original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, and performing sparse coding processing on the original image stream based on an event-driven mechanism to generate a dynamic perception event stream;

[0007] Based on the dynamic perception event stream, performing target detection and trajectory prediction processing through a sensing and computing integrated processing unit integrated in the chip to generate a vector data set including spatio-temporal correlation markers;

[0008] Based on the vector data set, performing heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline to generate a superimposed image with fused augmented reality annotations;

[0009] Performing adaptive transparency blending processing on the superimposed image, intermediate feature layer, and environmental state layer to generate a feedback control signal and a visual synthesis image;

[0010] Performing synchronous output processing on the visual synthesis image and the feedback control signal through a photon coding interface to obtain a synchronous output signal.

[0011] Further, based on the vector data set, heterogeneous graphics rendering processing is performed on the original image stream through a programmable rendering pipeline to generate a superimposed image with fused augmented reality annotations, including:

[0012] Based on the vector data set, channel-by-channel rendering processing is performed through a programmable shader in the programmable rendering pipeline to generate anti-aliased geometric graphics and photon diffraction textures. The programmable shader integrates a metasurface nanostructure, and the metasurface nanostructure performs phase control on incident light through wavefront modulation;

[0013] Based on the anti-aliased geometric graphics and photon diffraction textures, spatial mapping processing is performed on the three-dimensional space annotation information through a light field projection matrix to generate a superimposed image with fused augmented reality annotations.

[0014] Further, based on the vector data set, channel-by-channel rendering processing is performed through a programmable shader in the programmable rendering pipeline, and it also includes:

[0015] Wavefront modulation processing is performed on incident light through the metasurface nanostructure to generate a dynamically adjustable diffraction pattern.

[0016] Further, adaptive transparency blending processing is performed on the superimposed image, the intermediate feature layer, and the environmental state layer to generate a feedback control signal and a visual synthesis image, including:

[0017] Based on the non-linear relationship between the environmental light intensity and the target confidence level, dynamic transparency adjustment processing is performed on the superimposed image, the intermediate feature layer, and the environmental state layer to generate a superimposed image with optimized transparency;

[0018] Reset instruction generation processing is performed based on the annotation conflict rate of the visual synthesis image to generate a reset instruction, and the feedback control signal includes the reset instruction.

[0019] Further, synchronous output processing is performed on the visual synthesis image and the feedback control signal through a photon encoding interface to obtain a synchronous output signal, including:

[0020] Photon polarization state sequence conversion processing is performed on the visual synthesis image to generate a photon polarization state sequence;

[0021] Based on the photon polarization state sequence and the timestamp alignment mechanism, synchronous processing is performed on the feedback control signal and the image stream output to generate a synchronous output signal.

[0022] Further, the original image stream is collected through a complementary metal oxide semiconductor sensor with a bionic retina architecture, and sparse coding processing is performed on the original image stream based on an event-driven mechanism to generate a dynamic perception event stream, including:

[0023] Based on the biological retina pulse triggering mechanism, perform event coordinate and timestamp generation processing on the areas in the original image stream where the pixel brightness change exceeds a preset threshold to generate event coordinate and timestamp data;

[0024] Perform filtering processing on the event coordinate and timestamp data through spatio-temporal correlation verification to generate a continuous motion event stream;

[0025] Perform differential pulse signal encoding processing on the continuous motion event stream to generate a compressed event stream, and the dynamic perception event stream includes the compressed event stream.

[0026] Furthermore, based on the dynamic perception event stream, perform target detection and trajectory prediction processing through the integrated sensing and computing unit integrated in the chip to generate a vector data set including spatio-temporal association markers, including:

[0027] Based on the dynamic perception event stream, perform spatio-temporal feature extraction processing through a pulse convolution kernel to generate a target boundary pulse sequence;

[0028] Based on the target boundary pulse sequence, perform time integration processing through a neuromorphic computing unit to generate a motion trajectory prediction equation;

[0029] Based on the deviation amount between the motion trajectory prediction equation and the preset safety policy, perform dynamic alarm identification and control instruction generation processing to generate dynamic alarm identification and control instructions;

[0030] Generate a vector data set based on the dynamic alarm identification and control instructions.

[0031] In a second aspect, the present application also provides a monitoring system for visual images, and the system includes:

[0032] An event-driven perception module, configured to collect an original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, and perform sparse coding processing on the original image stream based on an event-driven mechanism to generate a dynamic perception event stream;

[0033] A sensing and computing fusion processing module, configured to perform target detection and trajectory prediction processing through the integrated sensing and computing unit integrated in the chip based on the dynamic perception event stream to generate a vector data set including spatio-temporal association markers;

[0034] A heterogeneous rendering engine module, configured to perform heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline based on the vector data set to generate a superimposed image with fused augmented reality annotations;

[0035] An adaptive fusion control module, configured to perform adaptive transparency blending processing on the superimposed image, the intermediate feature layer, and the environmental state layer to generate a feedback control signal and a visual synthesis image;

[0036] A photon closed-loop interface module is used to synchronously output and process a visualized synthetic image and a feedback control signal through a photon coding interface to obtain a synchronous output signal.

[0037] In a third aspect, the present application also provides a computer device, including a memory and an electronic chip. The memory stores a computer program, and when the processor executes the computer program, the steps of any method in the first aspect of the present application are implemented.

[0038] In a fourth aspect, the present application also provides an electronic chip, on which a computer program is stored. It is characterized in that when the computer program is executed by a processor, the steps of any method in the first aspect of the present application are implemented.

[0039] The technical solutions provided by the present application include the following technical effects: By providing a method for monitoring visual images, including: collecting an original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, sparsely encoding the original image stream based on an event-driven mechanism to generate a dynamic perception event stream; based on the dynamic perception event stream, performing target detection and trajectory prediction processing through a sensing and computing integrated processing unit integrated in the chip to generate a vector data set including spatio-temporal correlation tags; based on the vector data set, performing heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline to generate an overlay image with fused augmented reality annotations; performing adaptive transparency blending processing on the overlay image, intermediate feature layer, and environmental state layer to generate a feedback control signal and a visualized synthetic image; synchronously outputting and processing the visualized synthetic image and the feedback control signal through a photon coding interface to obtain a synchronous output signal, so as to solve the problem in the related art that dynamic transparency blending and photon-level low-latency output cannot be supported, and meet the requirements for high-precision real-time monitoring and closed-loop control in industrial scenarios. Description of the Drawings

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0041] Figure 1 It is a flowchart of a method for monitoring a visual image in an embodiment of the present invention;

[0042] Figure 2 It is a structural diagram of a monitoring system for a visual image in an embodiment of the present invention. Detailed Embodiments

[0043] To make the objectives, technical solutions, and advantages of this application clearer and more understandable, the following further elaborates on this application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0044] As Figure 1 shown, this application provides a method for monitoring visual images, including:

[0045] S101: Collect the original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, and perform sparse coding processing on the original image stream based on the event-driven mechanism to generate a dynamic perception event stream.

[0046] Specifically, a complementary metal oxide semiconductor (CMOS) sensor with a bionic retina architecture is used to collect the original image stream. This sensor mimics the working mechanism of the human eye retina and can efficiently sense light changes and convert them into electrical signals to achieve image acquisition. Each pixel unit in the sensor works independently. When the light intensity received by the pixel changes, an electrical signal is generated, and this signal is regarded as an "event". The event information generated by these pixels is recorded to form the original image stream. The original image stream is processed based on the event-driven mechanism. Different from the traditional frame scanning method, the event-driven mechanism is a real-time working asynchronous system that only triggers events when the light intensity of the pixel changes, thus avoiding repeated recording of static information. The event information in the original image stream is sparsely encoded.

[0047] Specifically, representative key events are extracted from a large number of pixel events, and redundant information is removed to generate a sparse event stream. This process can reduce the data volume and improve the processing efficiency. The event stream after sparse coding processing can more accurately reflect the dynamic changes in the image, such as the movement of objects, light changes, etc. These dynamic perception event streams can provide more effective data support for subsequent visual analysis tasks. Through the above steps, visual image information can be efficiently collected and processed, highlighting dynamic changes, and providing strong support for real-time monitoring and decision-making in scenarios such as industrial inspection and automation control.

[0048] S102: Based on the dynamic perception event stream, perform target detection and trajectory prediction processing through the integrated sensing and computing processing unit in the chip to generate a vector data set including spatio-temporal correlation tags.

[0049] Specifically, by using the integrated sensing and computing processing unit in the chip, the data in the dynamic sensing event stream is used as input and processed through a lightweight object detection network. This network structure module detects the location coordinates and class probabilities of the objects from the input event stream data. When designing the object detection network, the characteristic parameters of the memristors in the sensing and computing processing unit, such as resistance state instability, etc., are considered to ensure that the network can adapt to the non-ideal characteristics of the hardware. For the detected objects, the memory and computing synaptic array in the sensing and computing processing unit, combined with the long-term and short-term plasticity mechanism, is used to predict the movement trajectories of the objects. The memory and computing synaptic array can achieve in-situ calculation of historical input and current input, so as to quickly and accurately predict the positions of the objects at future moments. The results of object detection and trajectory prediction are integrated into a vector data set. During the process of generating the vector data set, spatio-temporal correlation tags are added to each object and its trajectory. The spatio-temporal correlation tags are used to identify the positions and motion states of the objects at different time points, so that the dynamic changes of the objects can be accurately correlated and tracked in subsequent analysis and processing.

[0050] S103: Based on the vector data set, the original image stream is subjected to heterogeneous graphics rendering processing through a programmable rendering pipeline to generate a superimposed image with fused augmented reality annotations.

[0051] Specifically, first, the generated vector data set is preprocessed to extract key information such as the positions, classes, and trajectories of the objects. This information will serve as the basis for subsequent rendering processing. Then, the programmable rendering pipeline is configured, including vertex shaders, fragment shaders, etc., to meet the requirements of heterogeneous graphics rendering. The vertex shader is responsible for processing the position information of the objects, and the fragment shader is used to process the appearance and augmented reality annotations of the objects. The programmable rendering pipeline is used to perform heterogeneous graphics rendering on the original image stream and the vector data set. On the one hand, basic image processing is performed on the original image stream, such as color correction, edge enhancement, etc.; on the other hand, according to the information in the vector data set, augmented reality annotations are added to the objects, such as drawing bounding boxes, class labels, trajectory lines, etc. of the objects. The augmented reality annotations are fused with the original image stream to generate a superimposed image. Through techniques such as transparency blending, it is ensured that the augmented reality annotations are naturally fused with the original image, so that the annotation information is clearly visible and does not block important image content. Further optimization processing is performed on the generated superimposed image, such as anti-aliasing, lighting consistency adjustment, etc., to improve the quality and visual effect of the image. Finally, the optimized superimposed image is output for subsequent display or analysis.

[0052] S104: Adaptive transparency blending processing is performed on the superimposed image, the intermediate feature layer, and the environmental state layer to generate a feedback control signal and a visual synthesis image.

[0053] Specifically, first, ensure that the data of the superimposed image, the intermediate feature layer, and the environmental state layer are ready. The superimposed image may be a pre-processed image, the intermediate feature layer can be the feature information extracted from a deep learning model, and the environmental state layer contains other relevant information about the environment, such as light, temperature, etc. Perform adaptive transparency blending on the superimposed image, the intermediate feature layer, and the environmental state layer. This usually involves the alpha blending algorithm, which fuses the information of different layers by adjusting the transparency (alpha value) of each pixel. Adaptive transparency blending can dynamically adjust the transparency parameters according to the environmental state or system requirements to achieve the best visual effect and information fusion. During the blending process, the system generates feedback control signals based on the blended image effect and the environmental state. These signals can be used to adjust system parameters, such as the sensitivity of sensors, the resource allocation of processing units, etc., to optimize subsequent image processing and system performance. Integrate the image after adaptive transparency blending to generate the final visual composite image.

[0054] S105: Synchronously output process the visual composite image and the feedback control signal through the photon encoding interface to obtain a synchronous output signal.

[0055] Specifically, first, ensure that the photon encoding interface has been correctly configured and initialized, including setting parameters such as the working mode, data transmission rate, and synchronization signal of the interface to ensure its effective communication with subsequent display devices or other processing units. Take the visual composite image and the feedback control signal as input data and send them into the photon encoding interface. Before input, it may be necessary to preprocess the data, such as data format conversion, data compression, etc., to meet the requirements of the photon encoding interface. The photon encoding interface generates a synchronization signal according to the input data and system requirements. The synchronization signal is used to ensure the synchronization of the visual composite image and the feedback control signal during output, avoiding data loss or abnormal display. Use the photon encoder in the photon encoding interface to convert the input electrical signal into an optical signal. During this process, the photon encoder encodes the data, such as pulse position modulation (PPM), etc., to improve the efficiency and reliability of data transmission. Perform synchronous output processing on the photon-encoded image data and the feedback control signal. The above steps involve precisely aligning two or more signals in time to ensure that they can reach the receiving end simultaneously and are correctly decoded and displayed at the receiving end. Transmit the synchronously output optical signal to the receiving end through an optical fiber or other optical transmission medium. At the receiving end, use the corresponding photon decoder to convert the optical signal back into an electrical signal for subsequent processing and display.

[0056] An embodiment of the present application provides a method for monitoring visual images, including: collecting an original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, performing sparse coding processing on the original image stream based on an event-driven mechanism to generate a dynamic perception event stream; based on the dynamic perception event stream, performing target detection and trajectory prediction processing through a computing-integrated processing unit integrated in the chip to generate a vector data set including spatio-temporal correlation tags; based on the vector data set, performing heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline to generate a superimposed image with fused augmented reality annotations; performing adaptive transparency blending processing on the superimposed image, intermediate feature layer, and environmental state layer to generate a feedback control signal and a visualized composite image; performing synchronous output processing on the visualized composite image and the feedback control signal through a photon coding interface to obtain a synchronous output signal, so as to solve the problem in the related art that dynamic transparency blending and photon-level low-latency output cannot be supported, and meet the requirements of high-precision real-time monitoring and closed-loop control in industrial scenarios.

[0057] Further, based on the vector data set, performing heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline to generate a superimposed image with fused augmented reality annotations, including:

[0058] Based on the vector data set, performing channel-by-channel rendering processing through a programmable shader in the programmable rendering pipeline to generate anti-aliased geometric graphics and photon diffraction textures, where the programmable shader integrates a metasurface nanostructure, and the metasurface nanostructure performs phase control on incident light through wavefront modulation;

[0059] Based on the anti-aliased geometric graphics and photon diffraction textures, performing spatial mapping processing on three-dimensional space annotation information through a light field projection matrix to generate a superimposed image with fused augmented reality annotations.

[0060] Specifically, first, ensure that the vector data set is ready, which contains key information such as the location, category, and trajectory of the target. Preprocess the vector data set, such as data cleaning, format conversion, etc., to make it compatible with the programmable rendering pipeline. Configure the programmable rendering pipeline, including setting the vertex shader, fragment shader, etc., to meet the requirements of heterogeneous graphics rendering. Among them, the programmable shader needs to be set for channel-by-channel rendering processing to prepare for subsequent anti-aliased geometry and photon diffraction texture generation. Use the programmable shader to perform channel-by-channel rendering processing on the vector data set. During this process, the shader will process different channels of the image (such as the RGB channel) according to the information in the vector data set to generate anti-aliased geometry and photon diffraction texture. The anti-aliased geometry reduces the jagged effect at the image edges through specific algorithms (such as Lanczos interpolation) to improve the smoothness and visual quality of the image. The photon diffraction texture involves the simulation and processing of the diffraction effect of light to enhance the realism and three-dimensionality of the image. Integrate the metasurface nanostructure into the programmable shader. The above nanostructure can control the phase of the incident light through wavefront modulation. The wavefront modulation technology can precisely adjust the phase distribution of light, thereby realizing the control of optical effects such as focusing and scattering of light, and further optimizing the display effect of the image. Based on the generated anti-aliased geometry and photon diffraction texture, use the light field projection matrix to perform spatial mapping processing on the three-dimensional space annotation information. The light field projection matrix can map the two-dimensional image data into the three-dimensional space, taking into account factors such as depth and viewing angle in the space, so that the augmented reality annotation can accurately fuse with the real-world spatial environment. Through spatial mapping processing, the virtual augmented reality annotation information is correctly superimposed on the corresponding position of the original image stream to generate the final superimposed image with the fused augmented reality annotation.

[0061] Furthermore, based on the vector data set, through the programmable shader in the programmable rendering pipeline for channel-by-channel rendering processing, it also includes:

[0062] Perform wavefront modulation processing on the incident light through the metasurface nanostructure to generate a dynamically adjustable diffraction pattern.

[0063] Specifically, preprocess the vector data set (data cleaning, format conversion, etc.) to make it compatible with the programmable rendering pipeline. Configure the pipeline, set the vertex and fragment shaders to meet the heterogeneous graphics rendering requirements. Use the programmable shader to process the vector data set by channels (such as RGB channels) to generate anti-aliased geometric graphics (using algorithms such as Lanczos interpolation to reduce jaggedness) and photon diffraction textures (simulating light diffraction to enhance realism and three-dimensionality). The programmable shader integrates the metasurface nanostructure, controls the phase of the incident light through wavefront modulation, precisely adjusts the light phase distribution, and realizes optical effects such as focusing and scattering to optimize the display effect. The nanostructure can also dynamically adjust the phase modulation by changing its own geometric shape, material properties or external excitation to generate different diffraction patterns. The dynamic modulation of the metasurface nanostructure can change the shape, position and intensity of the diffraction pattern in real time, and generate optical effects such as dynamic focusing according to different scenarios and requirements. Based on the generated graphics and textures, use the light field projection matrix to map the annotation information space in the three-dimensional space, map the two-dimensional image data to the three-dimensional space, consider factors such as depth and perspective, fuse the augmented reality annotation with the real space, and superimpose the virtual annotation on the corresponding position of the original image stream to generate the final superimposed image.

[0064] Furthermore, perform adaptive transparency blending processing on the superimposed image, the intermediate feature layer and the environmental state layer to generate a feedback control signal and a visual synthesis image, including:

[0065] Based on the non-linear relationship between the environmental light intensity and the target confidence, perform dynamic transparency adjustment processing on the superimposed image, the intermediate feature layer and the environmental state layer to generate a superimposed image with optimized transparency;

[0066] Generate a reset instruction based on the annotation conflict rate of the visual synthesis image to generate a reset instruction, and the feedback control signal includes the reset instruction.

[0067] Specifically, first, preprocess the vector data set, such as data cleaning and format conversion, to make it compatible with the programmable rendering pipeline. Then configure the programmable rendering pipeline, set the vertex and fragment shaders, and perform channel-based rendering processing settings on the programmable shader. Use it to process different channels of the image (such as the RGB channel) according to the vector data set information to generate anti-aliased geometric shapes (using algorithms such as Lanczos interpolation to reduce jaggedness) and photon diffraction textures (simulating light diffraction to enhance realism and three-dimensionality). The programmable shader integrates a metasurface nanostructure that controls the phase of the incident light through wavefront modulation, adjusts the light phase distribution, and realizes optical effects such as focusing and scattering. Moreover, the nanostructure can dynamically adjust the phase modulation by changing its own conditions, generate different diffraction patterns, and real-time change the shape, position, and intensity of the diffraction patterns to meet the requirements of different scenarios. Finally, based on the generated graphics and textures, use the light field projection matrix to perform spatial mapping on the three-dimensional space annotation information, map the two-dimensional image data to the three-dimensional space, consider factors such as depth and viewing angle, fuse the augmented reality annotation with the real space, and superimpose the virtual annotation on the corresponding position of the original image stream to generate the final superimposed image with the fused augmented reality annotation.

[0068] Furthermore, synchronously output process the visual synthesis image and the feedback control signal through the photon encoding interface to obtain a synchronous output signal, including:

[0069] Perform photon polarization state sequence conversion processing on the visual synthesis image to generate a photon polarization state sequence;

[0070] Based on the photon polarization state sequence and the timestamp alignment mechanism, synchronously process the feedback control signal and the image stream output to generate a synchronous output signal.

[0071] Specifically, perform photon polarization state sequence conversion processing on the visualized synthetic image to generate a photon polarization state sequence. This step involves converting image data into an optical signal sequence with specific polarization states. The photon polarization state sequence conversion can be achieved by using optical devices such as polarization controllers and polarization modulators. These devices can adjust the polarization state of light according to the input image data to generate a series of optical signals with specific polarization characteristics. To ensure the synchronization of the feedback control signal and the image stream output, it is necessary to perform synchronization processing on both based on the photon polarization state sequence and the timestamp alignment mechanism. The timestamp alignment mechanism involves adding timestamp information to the feedback control signal and the image stream output so that precise time alignment can be performed at the receiving end according to the timestamps. At the sending end, an accurate timestamp is assigned to each photon polarization state sequence and the feedback control signal, and these timestamps are recorded when sent. At the receiving end, according to the timestamp information in the received photon polarization state sequence and the feedback control signal, the output times of both are adjusted to align them in time. After the above processing, a synchronized output signal is generated. The synchronized output signal contains the visualized synthetic image after photon polarization state sequence conversion and the feedback control signal after timestamp alignment. These signals are synchronized in time and can be sent to subsequent processing units or display devices simultaneously, ensuring the coordination and consistency of the system in terms of display and control.

[0072] Furthermore, collect the original image stream through a complementary metal-oxide-semiconductor sensor of the bionic retina architecture, and perform sparse coding processing on the original image stream based on the event-driven mechanism to generate a dynamic perception event stream, including:

[0073] Based on the biological retina pulse triggering mechanism, perform event coordinate and timestamp generation processing on the regions in the original image stream where the pixel brightness change exceeds a preset threshold to generate event coordinate and timestamp data;

[0074] Perform filtering processing on the event coordinate and timestamp data through spatio-temporal correlation verification to generate a continuous motion event stream;

[0075] Perform differential pulse signal coding processing on the continuous motion event stream to generate a compressed event stream, and the dynamic perception event stream includes the compressed event stream.

[0076] Specifically, for the regions in the original image stream where the pixel brightness change exceeds a preset threshold, the biological retina pulse trigger mechanism is used to generate event coordinates and timestamp data. When the light intensity change of a pixel exceeds the preset threshold, a timestamp is marked with microsecond-level resolution, and an asynchronous event stream is output. The event coordinates and timestamp data are filtered through spatio-temporal correlation verification to generate a continuous motion event stream. According to the occurrence position of the current event, it is detected whether there are adjacent valid events in its neighborhood whose timestamp difference is within a set range (δt). If the condition is met, the current event is retained; otherwise, it is excluded. The continuous motion event stream is processed by differential pulse signal coding to generate a compressed event stream, and finally a dynamic perception event stream is formed. This coding method can further reduce the data volume and improve the efficiency of data transmission and processing.

[0077] Furthermore, based on the dynamic perception event stream, target detection and trajectory prediction processing are performed through the integrated sensing and computing unit integrated in the chip to generate a vector data set including spatio-temporal correlation marks, including:

[0078] Based on the dynamic perception event stream, spatio-temporal feature extraction processing is performed through a pulse convolution kernel to generate a target boundary pulse sequence;

[0079] Based on the target boundary pulse sequence, time integration processing is performed through a neuromorphic computing unit to generate a motion trajectory prediction equation;

[0080] Based on the deviation amount between the motion trajectory prediction equation and the preset safety policy, dynamic alarm marking and control instruction generation processing are performed to generate dynamic alarm marking and control instructions;

[0081] Based on the dynamic alarm marking and control instructions, a vector data set is generated.

[0082] Specifically, based on the dynamic perception event stream, spatio-temporal feature extraction processing is performed through a pulse convolution kernel to generate a target boundary pulse sequence. The pulse convolution kernel can capture the distribution characteristics of events in time and space, so as to accurately identify the boundary of the target. Based on the target boundary pulse sequence, time integration processing is performed through a neuromorphic computing unit to generate a motion trajectory prediction equation. The neuromorphic computing unit uses the characteristics of a spiking neural network (SNN) to integrate the pulse sequence, so as to predict the motion trajectory of the target. Based on the deviation amount between the motion trajectory prediction equation and the preset safety policy, dynamic alarm marking and control instruction generation processing are performed to generate dynamic alarm marking and control instructions. When the deviation between the predicted motion trajectory and the safety policy exceeds a certain threshold, the system will generate an alarm marking and corresponding control instructions to take necessary countermeasures. Based on the dynamic alarm marking and control instructions, a vector data set is generated. The vector data set contains the position, category, trajectory, and alarm information of the target, etc., and is represented in vector form for subsequent processing and analysis.

[0083] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the indications of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0084] In one embodiment, the present application further provides a monitoring system 200 for visual images, and the system includes:

[0085] An event-driven perception module 201, configured to collect an original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, and perform sparse coding processing on the original image stream based on an event-driven mechanism to generate a dynamic perception event stream;

[0086] A sensing and computing fusion processing module 202, configured to perform target detection and trajectory prediction processing on the basis of the dynamic perception event stream through a sensing and computing integrated processing unit integrated in the chip, and generate a vector data set including spatio-temporal association marks;

[0087] A heterogeneous rendering engine module 203, configured to perform heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline based on the vector data set, and generate a superimposed image with fused augmented reality annotations;

[0088] An adaptive fusion control module 204, configured to perform adaptive transparency mixing processing on the superimposed image, the intermediate feature layer, and the environmental state layer to generate a feedback control signal and a visual synthesis image;

[0089] A photon closed-loop interface module 205, configured to perform synchronous output processing on the visual synthesis image and the feedback control signal through a photon coding interface to obtain a synchronous output signal.

[0090] Specifically, the event-driven perception module 201 acquires the original image stream through a complementary metal oxide semiconductor sensor with a bionic retina architecture, and performs sparse coding processing based on the event-driven mechanism to generate a dynamic perception event stream. The sensing and computing fusion processing module 202 uses the integrated sensing and computing unit on the chip to perform target detection and trajectory prediction processing based on the dynamic perception event stream, and generates a vector data set containing spatio-temporal correlation tags. The heterogeneous rendering engine module 203 performs heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline based on the vector data set, and generates an overlay image with fused augmented reality annotations. The adaptive fusion control module 204 performs adaptive transparency blending processing on the overlay image, the intermediate feature layer, and the environmental state layer to generate a feedback control signal and a visual synthesis image. The photon closed-loop interface module 205 performs synchronous output processing on the visual synthesis image and the feedback control signal through a photon coding interface to obtain a synchronous output signal. Through the collaborative work of each module, the above technical solution realizes efficient processing from image acquisition to synchronous output, solves the problems in the related technology that cannot support dynamic transparency blending and photon-level low-latency output, and meets the requirements of high-precision real-time monitoring and closed-loop control in industrial scenarios.

[0091] The heterogeneous rendering engine module 203 is further configured to:

[0092] Based on the vector data set, perform channel-by-channel rendering processing through programmable shaders in the programmable rendering pipeline to generate anti-aliased geometric graphics and photon diffraction textures. The programmable shader integrates a metasurface nanostructure, and the metasurface nanostructure performs phase control on the incident light through wavefront modulation;

[0093] Based on the anti-aliased geometric graphics and photon diffraction textures, perform spatial mapping processing on the three-dimensional space annotation information through a light field projection matrix to generate an overlay image with fused augmented reality annotations.

[0094] The heterogeneous rendering engine module 203 is further configured to:

[0095] Perform wavefront modulation processing on the incident light through the metasurface nanostructure to generate a dynamically adjustable diffraction pattern.

[0096] The adaptive fusion control module 204 is further configured to:

[0097] Based on the non-linear relationship between the environmental light intensity and the target confidence level, perform dynamic transparency adjustment processing on the overlay image, the intermediate feature layer, and the environmental state layer to generate an overlay image with optimized transparency;

[0098] Generate a reset instruction generation process based on the annotation conflict rate of the visual synthesis image to generate a reset instruction, and the feedback control signal includes the reset instruction.

[0099] The photon closed-loop interface module 205 is further configured to:

[0100] Perform photon polarization state sequence conversion processing on the visualized synthetic image to generate a photon polarization state sequence;

[0101] Based on the photon polarization state sequence and the timestamp alignment mechanism, synchronize the feedback control signal and the image stream output to generate a synchronized output signal.

[0102] The event-driven perception module 201 is further configured to:

[0103] Based on the biological retina impulse triggering mechanism, perform event coordinate and timestamp generation processing on the area where the pixel brightness change in the original image stream exceeds a preset threshold to generate event coordinate and timestamp data;

[0104] Filter the event coordinate and timestamp data through spatio-temporal correlation verification to generate a continuous motion event stream;

[0105] Perform differential pulse signal encoding processing on the continuous motion event stream to generate a compressed event stream, and the dynamic perception event stream includes the compressed event stream.

[0106] The sensing and computing fusion processing module 202 is further configured to:

[0107] Based on the dynamic perception event stream, perform spatio-temporal feature extraction processing through a pulse convolution kernel to generate a target boundary pulse sequence;

[0108] Based on the target boundary pulse sequence, perform time integration processing through a neuromorphic computing unit to generate a motion trajectory prediction equation;

[0109] Based on the deviation amount between the motion trajectory prediction equation and the preset safety policy, perform dynamic warning identification and control instruction generation processing to generate dynamic warning identification and control instructions;

[0110] Generate a vector data set based on the dynamic warning identification and control instructions.

[0111] In one embodiment, the present application further provides a computer device, including a memory and an electronic chip. The memory stores a computer program, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0112] In one embodiment, the present application further provides an electronic chip, on which a computer program is stored, and characterized in that when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0113] In one embodiment, the chip uses a cold light source or any other applicable light source, which can be switched according to service requirements, and the light source remains in a constant on state.

[0114] Nanopores are processed on the substrate for light transmission to form dense light spots. The substrate can be independently selected according to the required material properties, and its main functions are light shielding and light transmission. The processing technology of the light transmission holes is not limited, and etching, nano-printing or other processes can be used as long as the light transmission requirements are met.

[0115] Regarding signal capture and output, domestic optoelectronic components currently adapted to light spot capture are selected to design a light spot capture component to perform the light spot scanning task. When light is scanned, the output signal is "1"; when no light is scanned, the output signal is "0", so as to realize the output of basic logic operation signals. In addition, the scanning can be carried out by adding color distinction to the light spots to increase the parallel computing channels, and any light technology owned by any country can be incorporated into the chip design concept at present, so as to comprehensively improve the comprehensive performance of the chip and meet the task requirements of different application scenarios. According to the above design, 1 light spot is a data processing channel, and many dense light spots are scanned synchronously, which can directly improve the parallel data processing ability. After integrating various light technologies, the improvement amplitude of the performance needs to be estimated through actual design.

[0116] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are only illustrative. The components described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the present disclosure solution. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0117] The above embodiments only represent several implementation manners of the embodiments of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the application embodiments. It should be noted that for those of ordinary skill in the art, without departing from the concept of the embodiments of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the embodiments of the present application.

Claims

1. A method for monitoring visual images, characterized in that: The method comprises: The original image stream is collected by a complementary metal oxide semiconductor sensor of a bionic retina architecture, and the original image stream is sparsely encoded based on an event-driven mechanism to generate a dynamic perception event stream; Based on the dynamic perception event stream, target detection and trajectory prediction processing are performed through the integrated sensing and computing processing unit in the chip to generate a vector data set including spatiotemporal association tags; Based on the vector data set, performing heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline to generate a superimposed image fused with augmented reality annotations; Performing adaptive transparency mixing processing on the superimposed image, the intermediate feature layer and the environmental state layer to generate a feedback control signal and a visual composite image; The visualized synthetic image and the feedback control signal are synchronously output processed through a photon coding interface to obtain a synchronous output signal.

2. A visual image monitoring method according to claim 1, characterized in that: The method of performing heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline based on the vector data set to generate a superimposed image fused with augmented reality annotations includes: Based on the vector data set, performing channel rendering processing by a programmable shader in the programmable rendering pipeline to generate anti-aliased geometric figures and photon diffraction textures, wherein the programmable shader integrates a metasurface nanostructure, and the metasurface nanostructure performs phase control on incident light by wavefront modulation; Based on the anti-aliasing geometric figures and the photon diffraction texture, the three-dimensional space annotation information is spatially mapped through a light field projection matrix to generate the superimposed image fused with augmented reality annotations.

3. A visual image monitoring method according to claim 2, characterized in that: The performing channel rendering processing based on the vector data set by a programmable shader in the programmable rendering pipeline further includes: The incident light is wavefront modulated by the metasurface nanostructure to generate a dynamically adjustable diffraction pattern.

4. A visual image monitoring method according to claim 1, characterized in that: The step of performing adaptive transparency mixing processing on the superimposed image, the intermediate feature layer and the environmental state layer to generate a feedback control signal and a visual composite image includes: Based on the nonlinear relationship between the ambient light intensity and the target confidence, dynamically adjusting the transparency of the superimposed image, the intermediate feature layer and the environmental state layer to generate a superimposed image with optimized transparency; A reset instruction generation process is performed based on the annotation conflict rate of the visualized synthetic image to generate a reset instruction, and the feedback control signal includes the reset instruction.

5. The visual image monitoring method according to claim 1, characterized in that: The step of performing synchronous output processing on the visualized synthetic image and the feedback control signal through a photon coding interface to obtain a synchronous output signal includes: Performing photon polarization state sequence conversion processing on the visualized synthetic image to generate a photon polarization state sequence; The feedback control signal and the image stream output are synchronously processed based on the photon polarization state sequence and the timestamp alignment mechanism to generate the synchronous output signal.

6. A visual image monitoring method according to claim 1, characterized in that: The method collects the original image stream through the complementary metal oxide semiconductor sensor of the bionic retina architecture, performs sparse coding processing on the original image stream based on the event-driven mechanism, and generates a dynamic perception event stream, including: Based on the biological retinal pulse trigger mechanism, event coordinate and timestamp generation processing is performed on the area in the original image stream where the pixel brightness change exceeds a preset threshold, so as to generate event coordinate and timestamp data; Filtering the event coordinates and timestamp data through spatiotemporal correlation verification to generate a continuous motion event stream; The continuous motion event stream is subjected to differential pulse signal encoding processing to generate a compressed event stream, and the dynamic perception event stream includes the compressed event stream.

7. The visual image monitoring method according to claim 1, characterized in that: Based on the dynamic perception event stream, the target detection and trajectory prediction processing are performed by the integrated sensing and computing processing unit in the chip to generate a vector data set including spatiotemporal association tags, including: Based on the dynamic perception event stream, a spatiotemporal feature extraction process is performed through a pulse convolution kernel to generate a target boundary pulse sequence; Based on the target boundary pulse sequence, a time integration process is performed by a neuromorphic computing unit to generate a motion trajectory prediction equation; Based on the deviation between the motion trajectory prediction equation and the preset safety strategy, a dynamic warning mark and a control instruction generation process are performed to generate a dynamic warning mark and a control instruction; The vector data set is generated based on the dynamic alarm identification and the control instruction.

8. A visual image monitoring system, characterized in that: The system comprises: An event-driven perception module is used to collect raw image streams through complementary metal oxide semiconductor sensors of a bionic retina architecture, and to perform sparse coding processing on the raw image streams based on an event-driven mechanism to generate dynamic perception event streams; A sensing and computing fusion processing module is used to perform target detection and trajectory prediction processing based on the dynamic perception event stream through the sensing and computing integrated processing unit integrated in the chip to generate a vector data set including spatiotemporal association tags; A heterogeneous rendering engine module, configured to perform heterogeneous graphics rendering processing on the original image stream through a programmable rendering pipeline based on the vector data set to generate a superimposed image fused with augmented reality annotations; An adaptive fusion control module, used for performing adaptive transparency mixing processing on the superimposed image, the intermediate feature layer and the environmental state layer to generate a feedback control signal and a visual composite image; The photon closed-loop interface module is used to perform synchronous output processing on the visualized synthetic image and the feedback control signal through a photon coding interface to obtain a synchronous output signal.

9. A computer device comprising a memory and an electronic chip, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of a visual image monitoring method described in any one of claims 1 to 7 are implemented.

10. An electronic chip having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a visual image monitoring method according to any one of claims 1 to 7 are implemented.

Images