An imaging control method based on retinal imaging principle

Through the hierarchical downsampling and decision-making network technology of the principle of retinal imaging, the problem of high consumption of traditional camera storage and computing resources is solved, and the accuracy and confidence of edge object recognition is improved, achieving efficient image processing and recognition effects.

CN118574022BActive Publication Date: 2024-11-12NANJING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411047663.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-12
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Traditional cameras require a lot of storage space and computing resources during imaging, and the resolution at the edges is too low, so it is impossible to accurately determine the identity of objects at the edge of the field of view, and it is impossible to dynamically adjust the center position of the imaging field of view to improve recognition confidence.

Method used

Using a hierarchical downsampling strategy and decision network based on the principle of retinal imaging, the pixel position is calculated through the isomorphic row and column site selection circuit, the image is graded downsampled, and the object information is analyzed through the decision network, and the next frame of image is dynamically adjusted. Sampling strategy, using Transformer model to implement loss function to improve object recognition accuracy.

Benefits of technology

While ensuring the resolution of the focus, the overall image resolution is reduced, the storage space and calculation amount is saved, the confidence in object recognition in the field of view is improved, and the data transmission bandwidth and power consumption are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118574022B_ABST
    Figure CN118574022B_ABST
Patent Text Reader

Abstract

The present invention discloses an imaging control method based on the principle of retinal imaging, which belongs to the field of imaging technology. The method adopts a hierarchical downsampling strategy to simulate the characteristics of retinal imaging, while ensuring the resolution of the focus point, reducing the resolution of the overall image, which can greatly reduce the data storage space, and the decision network evaluates and dynamically observes the output results of the imaging method, and comprehensively considers the estimated loss of the identity, position and size of the objects in the scene, as well as the estimated loss of the observation results at the next moment and the difference between the generation probability and the posterior probability of the decision network. The estimated loss, so as to accurately understand the scene at the current moment and determine the hierarchical downsampling strategy at the next moment, and in network calculation, the method can greatly reduce the amount of calculation, effectively reduce the data transmission bandwidth and power consumption, and provide the possibility of application to edge devices.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to an imaging control method based on the retinal imaging principle, belonging to the technical field of imaging. Background Art

[0002] As a photoelectric signal conversion device, the camera can effectively collect light signals in the environment and convert them into electrical signals to display images. It is widely used in daily life, security, national defense and other fields. In traditional cameras, the imaging chip reads out all pixels evenly, that is, all pixels are processed equally. Although this processing method can obtain high-quality images, it requires a lot of storage space and requires huge computing power and power consumption for subsequent image processing tasks.

[0003] Unlike traditional cameras, where imaging chips uniformly collect light signals from the environment, biological objects, such as the human retina, sample non-uniformly during imaging, i.e., the image is sampled at a high resolution at the center of vision and at a low resolution at the edge of vision. If the camera can imitate the retinal imaging principle to achieve non-uniform sampling, it can reduce the requirements for computing power and power consumption as much as possible while meeting the resolution requirements.

[0004] When obtaining imaging results by simulating the principle of retinal imaging, another issue that needs to be considered is that due to the low resolution at the edge, it will be impossible to accurately judge the identity of objects at the edge of the field of view. Therefore, it is necessary to explore targeted improvement methods to achieve accurate estimation of low-resolution images at the edge, and to be able to dynamically adjust the center position of the imaging field of view to improve the confidence in the identification of each object in the field of view. Summary of the invention

[0005] In order to solve the existing problems, the present invention provides an imaging control method based on the principle of retinal imaging. Through a hierarchical downsampling strategy, the characteristics of retinal imaging are simulated to reduce the resolution of the overall image while ensuring the resolution of the focus point. The imaging control method based on the principle of retinal imaging has an isomorphic row addressing circuit and a column addressing circuit. The row addressing circuit and the column addressing circuit inputs include the center coordinates of the focus point, the resolution size of the focus area, and the downsampling level n. The pixel position to be selected is calculated based on the three inputs, and is read out and quantized through a readout circuit. At the same time, the imaging control method based on the principle of retinal imaging has an n-level storage area corresponding to the n-level downsampling, that is, the result of each level of downsampling readout is stored in the access area corresponding to the first level.

[0006] Specifically, the imaging control method based on the retinal imaging principle provided in the present application is applied to a retinal imaging camera, wherein the retinal imaging camera is provided with a pixel array, a row addressing circuit, a column addressing circuit, a readout circuit, and n storage areas; the method adopts a hierarchical downsampling strategy to obtain the sampling images of each level corresponding to the current frame image, and analyzes the object information in each level of the image through a decision network, and determines the hierarchical downsampling strategy of the next frame image according to the object information, wherein the hierarchical downsampling strategy includes the center point coordinates , the size of each level of sampling image , and the currently adopted downsampling level n; the imaging chip of the retinal imaging camera selects the corresponding pixels in the pixel array for sampling according to the graded downsampling strategy.

[0007] This method uses a hierarchical downsampling strategy to obtain the sampling images at each level corresponding to the next frame image. The downsampled image is in the sampling range of Every 2 n-1 The size of the sample obtained by sampling one point is of the image.

[0008] Furthermore, when the method performs hierarchical downsampling on an image, if the sampling range exceeds the range of the pixel array, zeros are padded to the sampling points that exceed the range.

[0009] Furthermore, the decision network is implemented based on the Transformer model, and the loss function includes an estimated function term for the identity of each object in the entire scene. , the estimated function term for the position of each object in the entire scene , the estimated function term for the size of each object in the entire scene , the estimated function term for the observation result at the next moment And the difference function term between the decision network generation probability and the posterior probability ; Loss function for:

[0010]

[0011] in, , , , , is the weight value corresponding to each function item.

[0012] Furthermore, the estimated function term for the identity of each object in the entire scene is for:

[0013]

[0014] in, represents the identity of the kth object in the entire scene, Decision Network The parameterized posterior probability distribution, represents the observation angle at time t, , , Respectively represent the first Objects At the moment identity, position and size in the class, represents the L2 norm.

[0015] Furthermore, the estimated function term for the position of each object in the entire scene is for:

[0016]

[0017] in, Represents the position of the kth object in the entire scene.

[0018] Furthermore, the estimated function term for the size of each object in the entire scene is for:

[0019]

[0020] in, Indicates the size of the kth object in the entire scene.

[0021] Furthermore, the estimated function term for the observation result at the next moment for:

[0022]

[0023] in, represents the observation at time t+1 predicted by the decision network, Decision Network Parameterized generative probability distribution, represents the observation perspective at time t+1 predicted by the decision network, , , They represent the first The object at time t+1 is Identity, position and size in the hierarchy.

[0024] Furthermore, the difference function term between the generation probability and the posterior probability of the decision network is for:

[0025]

[0026] in, represents the KL divergence, which is used to measure the difference between two probability distributions.

[0027] The imaging control method based on the retinal imaging principle proposed in the present invention is compared with the typical camera imaging method. The imaging method adopts a sampling interval of 1 for sampling the first-level downsampling area, a sampling interval of 2 for sampling the second-level downsampling area, a sampling interval of 4 for sampling the third-level downsampling area, and so on. The sampling interval of 2 for sampling the nth-level downsampling area is used. n-1 The sampling interval is 100, and the resulting n pictures of different resolutions have a data storage capacity reduced by n / 2 compared to the full sampling of the whole picture. n-1 , saving a lot of storage space and alleviating the amount of computation required for image processing. Compared with ordinary cameras, the amount of computation is reduced by n / 4 n-1 .

[0028] The imaging control method based on the retinal imaging principle proposed in the present invention uses a decision network to analyze the identity, size, position and other information of objects in each level of the image for the multi-level image obtained by using the imaging method, and estimates the distribution of each object in the original image in the entire scene. The imaging strategy at the next moment is dynamically adjusted based on the evaluation result, that is, how the focus center, size and number of levels of the next imaging should be configured. The use of the decision network can effectively solve the problem of being unable to accurately judge the identity of objects at the edge of the visual field due to the low resolution at the edge, and can effectively improve the confidence of identifying each object in the visual field.

[0029] The beneficial effects of the present invention are:

[0030] The present application scheme adopts a hierarchical downsampling strategy to simulate the characteristics of retinal imaging, reducing the resolution of the overall image while ensuring the resolution of the focus point, which can greatly reduce data storage space. The decision network can evaluate and dynamically observe the output results of the imaging method, and in network computing, this method can greatly reduce the amount of calculation, effectively reduce data transmission bandwidth and power consumption, and is expected to become a standard deployment paradigm for edge imaging devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 Schematic diagram of hierarchical downsampling of an image while focusing on the center of the image.

[0033] Figure 2 Schematic diagram of the first level sampling of the image.

[0034] Figure 3 Schematic diagram of the second level sampling of the image.

[0035] Figure 4 Schematic diagram of the third-level sampling of the image when focusing on the center of the image.

[0036] Figure 5 Schematic diagram of hierarchical downsampling of an image while focusing on the image edges.

[0037] Figure 6 Schematic diagram of the third level sampling of the image when focusing on the edge of the image.

[0038] Figure 7 Schematic diagram of the circuit structure of the retinal imaging camera.

[0039] Figure 8 Schematic diagram of the decision network for the output of the retinal imaging camera.

[0040] Fig. 9 Schematic diagram of the decision network for the output results of an ordinary camera.

[0041] Fig.10 Schematic diagram of the generative model for the decision network of a retinal imaging camera.

[0042] Fig.11 Schematic diagram of the inference process of the decision network for a retinal imaging camera.

[0043] Fig.12 Network model structure diagram for the retinal imaging camera decision network. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0045] Embodiment 1:

[0046] This embodiment provides an imaging control method based on the retinal imaging principle. The method is applied to a retinal imaging camera. The retinal imaging camera is provided with a pixel array, a row addressing circuit, a column addressing circuit, a readout circuit, and n storage areas. The imaging chip of the retinal imaging camera adopts a hierarchical downsampling strategy to obtain the current frame image, and analyzes the object information in each level of the image through a decision network, and determines the hierarchical downsampling strategy for the next frame image according to the object information. The object information includes the identity, size, position and other information of the object. The hierarchical downsampling strategy includes the coordinates of the center point , the size of each level of sampling image , and the currently adopted downsampling level n; the imaging chip of the retinal imaging camera selects the corresponding pixels in the pixel array for sampling according to the graded downsampling strategy.

[0047] like Figure 1 As shown in FIG. 1 , an example of three-level downsampling of an image is performed when the center of the gaze is the center of the image, where the size of each level of the image is The size of the image. Assume that the center position of the image sampling is , the image size of the first level sampling is , then the range of the first-level sampled image is , where the first item is the coordinate position of the upper left corner of the first-level sampling picture, and the second item is the coordinate position of the lower right corner of the first-level sampling picture. The origin of the coordinates is the upper left corner of the entire imaging array. Within this coordinate range, the picture is sampled at the first level, that is, each point is sampled. The resulting picture is as follows Figure 2 As shown; for the range of the second-level sampled picture, it is , within this coordinate range, the image is sampled twice, that is, sampling is performed every two points to ensure that the size of the final image is The resulting picture is Figure 3 As shown; for the range of the third-level sampled picture, it is , within this coordinate range, the image is sampled at three levels, that is, every four points are sampled once to ensure that the size of the final image is The resulting picture is Figure 4 As shown; when n-level downsampling is required, then by analogy, the range of the image at the n-th level of sampling is , within this coordinate range, the image is Level sampling, that is, every 2 n-1 The points are sampled once to ensure that the size of the final image is .

[0048] During the hierarchical downsampling process, if the sampling range exceeds the range of the pixel array, the sampling points that exceed the range are padded with zeros, such as Figure 5As shown in FIG. 1 , an example of three-level downsampling of an image when the center of attention is at the edge of the image, where the size of each level of image is 8×8. In this example, the first level sampling and the second level sampling are the same as Figure 1 The situation in the example is the same. For the third-level sampling, since the focus center is at the edge of the image, the range of the third-level sampling exceeds the boundary of the image. Figure 5 In the third level sampled image, some areas in the upper and left parts exceed the coordinate range of the pixel array. In this case, the sampling points beyond the coordinate range need to be padded with zeros. The third level sampled image is as follows: Figure 6 As shown, in this example, the first row and the first column of the third-level sampling picture are beyond the range of the pixel array, and the positions of the corresponding sampling points in the first row and the first column of the obtained third-level sampling picture are directly set to 0, and the remaining sampling points within the pixel array range are sampled normally.

[0049] like Figure 7 FIG. 1 is a schematic diagram of the circuit structure of a retinal imaging camera, in which the camera includes a pixel array, a row addressing circuit, a column addressing circuit, a readout circuit, and The inputs of the row addressing circuit and the column addressing circuit include the coordinates of the center point of attention. , the size of each level of pictures , and the current downsampling level , as mentioned above, for The image with downsampled level needs to be sampled Line and Column, sampling range is , the rows that need to be sampled are 2 n-2 2 n-1 , the columns that need to be sampled are 2 n-2 2 n-1 The row selection circuit and the column selection circuit receive the coordinates of the center of attention. , the size of each level of pictures , and the current downsampling level After the signal is received, the row addressing circuit will sequentially select the rows of the pixel array to be read, and at the same time, the column addressing circuit will select all the columns of the pixel array to be read. Level Line and The pixels of the column are read out and quantized by the readout circuit and then stored in the corresponding The size of each storage area is .

[0050] like Figure 8 As shown, it is a schematic diagram of the decision network structure for processing the output results of the retinal imaging camera. At the initial moment, the decision network uses the initial focus point center coordinates (taking the exact center of the image as an example), the focus area resolution size (taking the size of one sixteenth of the overall image resolution as an example), and the downsampling level (taking five levels as an example) to obtain a non-uniformly sampled image, and inputs the non-uniformly sampled image into the decision network. The output result of the decision network is the focus point center coordinates of the next frame image. , focus on the regional resolution size , downsampling level , thereby dynamically adjusting the next frame of image and realizing a retinal-like scanning process. In the decision network, the input is Size In image processing, the decision network usually uses a convolutional neural network. If the first layer of convolution in the convolutional neural network has channels, and the convolution kernel size of each channel is , the stride of the convolution kernel is , fill in the edges of the input image Circle zero. In the convolution operation, the number of multiplications required is: , the number of additions required is: , and if Fig. 9 The decision network diagram of the output result of a common camera is shown in Figure 1. That is, when all pixels in the pixel array are read out equally, the input of the decision network is a 2-pixel image. n-1 2 n-1 , then in the convolutional neural network, the first layer of convolution has channels, and the convolution kernel size of each channel is , the stride of the convolution kernel is , fill in the edges of the input image Circle zero. In the convolution operation, the number of multiplications required is: {[(2 n-1 ) / ]+1} 2 , the number of additions required is: ( ) {[(2 n-1 ) / ]+1} 2 For the computational complexity of the decision network of retinal imaging cameras and ordinary cameras, in the same decision network structure, taking a layer of convolution calculation as an example, since the computational complexity is mainly related to the image size, the number of multiplication calculations can be reduced by approximately n / 4 n -1 , the number of addition calculations can be approximately reduced ( ) / [( ) 4 n-1 ], since the convolution calculation usually only considers multiplication operations, it can be roughly regarded that the calculation amount of the retinal imaging camera in the decision network is reduced by n / 4 compared with the ordinary camera n-1 .

[0051] The decision network required for the retinal imaging camera can be summarized as follows: Fig.10 The generative model shown in this model, for the scene , if exists Objects , for each object, its identity is , and its position in the entire image is , whose size is , and the camera is The action at the moment is , the meaning of this action is Always adjust the focus of the retinal imaging camera to , and divided into The size of each level is , thus forming a new perspective on the scene , observing the scene from this perspective, we get objects, in At the moment, Level status ,Location ,size , which formed the retinal imaging camera in Observation of the moment The joint probability distribution of this generative model can be described as formula (1):

[0052] (1)

[0053] in For The hidden state of the system at all times, including identity ,Location ,size And other variables describing the system state. Introduce the posterior probability of the model , then the free energy of the model is as shown in formula (2), and the goal of this model is to minimize the difference between the joint probability distribution of the model prediction and the generated model, that is, to minimize the free energy , as shown in formula (2).

[0054] (2)

[0055] To implement the decision network, the inference process is as follows Fig.11 As shown, for each observation, the output of the retinal imaging camera , the decision network first determines objects, in At the moment, Level status ,Location ,size , and according to the current observation angle , infer the identity of each object in the entire scene , the position in the whole picture , and size , and select the next observation angle based on the inference result , and the network will predict the first objects, in At the moment, Level status ,Location ,size , which predicts Observation of the moment The decision network described in this way is trained, and its loss function consists of the following five parts:

[0056] (1) Estimation function for the identity of each object in the entire scene:

[0057] (3)

[0058] (2) Estimation function for the position of each object in the entire scene:

[0059] (4)

[0060] (3) Estimation function for the size of each object in the entire scene:

[0061] (5)

[0062] (4) Estimation function term for the observation result at the next moment:

[0063] (6)

[0064] (5) The difference function term between the generated probability and the posterior probability of the decision network:

[0065] (7)

[0066] in, Decision Network The parameterized posterior probability distribution, Decision Network Parameterized generative probability distribution, represents the L2 norm, represents the KL divergence, which is used to measure the difference between two probability distributions.

[0067] Combining the above five formulas, the total loss function of the model is:

[0068] (8)

[0069] in, , , , , is the weight value corresponding to each function item.

[0070] In this model, the purpose of the scene estimation generated by each observation is to provide guidance for the next observation, that is, to better observe the target and estimate the scene. The strategy involved in the selection of observation perspective is:

[0071] (9)

[0072] In equation (9), the first term on the right side of the equation represents the correct identification of the target object, the second term represents the correct location of the target object, the third term represents the correct size of the target object, and the fourth and fifth terms represent the continuous search for the target object. , hoping to make The value of is the smallest, so we can use Monte Carlo sampling to randomly sample formula (9) to find out The candidate perspective with the smallest value .

[0073] For the decision network, the following can be used: Fig.12The Transformer model shown in the figure is implemented. The model is trained using the ImageNet dataset. The non-uniform images sampled by the retinal imaging camera are simulated by data preprocessing. For multiple images obtained through non-uniform sampling, the images are serialized and position-encoded by patching. The obtained vectors are passed through the transformer encoding module to obtain the entire scene. Estimates of the identity of each object in the entire scene , the position in the whole picture , and size , and then input the estimate into the transformer decoding module to predict how the camera should observe the image at the next moment, and estimate the first objects, in At the moment, Level status ,Location ,size , which predicts Observation of the moment .

[0074] The imagenet dataset used in this application is a public image dataset that contains more than 14 million images, covering more than 20,000 categories, of which more than one million images have clear category annotations and annotations of the location of objects in the image. The images in the ImageNet dataset cover most of the image categories that you will see in life, and each image is manually labeled with a category to ensure the quality and diversity of the dataset.

[0075] Some steps in the embodiments of the present invention may be implemented using software, and the corresponding software program may be stored in a readable storage medium, such as a CD or a hard disk.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An imaging control method based on retinal imaging principle, characterized in that: The method is applied to a retinal imaging camera, which is provided with a pixel array, a row addressing circuit, a column addressing circuit, a readout circuit, and n storage areas; the method adopts a hierarchical downsampling strategy to obtain sampling images of each level corresponding to the current frame image, and analyzes the object information in each level of the image through a decision network, and determines the hierarchical downsampling strategy for the next frame image according to the object information, wherein the hierarchical downsampling strategy includes the center point coordinates , the size of each level of sampling image , and the currently adopted downsampling level n; the imaging chip of the retinal imaging camera selects the corresponding pixels in the pixel array for sampling according to the graded downsampling strategy; When the method adopts a hierarchical downsampling strategy to obtain the sampling images of each level corresponding to the next frame image, the first level downsampling image is The image of size s×s is obtained by sampling each point in the area. The image downsampled by level (n≥2) is in the sampling range of Every 2 n-1 The size of the sample obtained by sampling one point is images; The decision network is implemented based on the Transformer model. In the model, for the scene , if exists Objects , for each object, its identity is , and its position in the entire image is , whose size is , and the camera is The action at the moment is , the meaning of this action is Always adjust the focus of the retinal imaging camera to , and divided into The size of each level is , thus forming a new perspective on the scene , observing the scene from this perspective, we get objects, in At the moment, Level status ,Location ,size , which formed the retinal imaging camera in Observation of the moment ; Its loss function includes an estimated function term for the identity of each object in the entire scene , the estimated function term for the position of each object in the entire scene , the estimated function term for the size of each object in the entire scene , the estimated function term for the observation result at the next moment And the difference function term between the decision network generation probability and the posterior probability ; Loss function for: in, , , , , is the weight value corresponding to each function item; The estimated function term for the identity of each object in the entire scene for: in, represents the identity of the kth object in the entire scene, Decision Network The parameterized posterior probability distribution, represents the observation angle at time t, , , Respectively represent the first Objects At the moment identity, position and size in the class, represents the L2 norm; The estimated function term for the position of each object in the entire scene for: in, Indicates the position of the kth object in the entire scene; The estimated function term for the size of each object in the entire scene for: in, Indicates the size of the kth object in the entire scene; The estimated function term for the observation result at the next moment for: in, represents the observation at time t+1 predicted by the decision network, Decision Network Parameterized generative probability distribution, represents the observation perspective at time t+1 predicted by the decision network, , , They represent the first The object at time t+1 is identity, position and size in the hierarchy; The difference function term between the generated probability and the posterior probability of the decision network for: in, represents the KL divergence, which is used to measure the difference between two probability distributions.

2. The method according to claim 1, characterized in that: When the method performs hierarchical downsampling on an image, if the sampling range exceeds the range of the pixel array, zero padding is performed on the sampling points that exceed the range.