CCD image processing method and device and storage medium

Abstract

The invention discloses a CCD image processing method and device and a storage medium. In the method, a server can split to-be-processed image data into to-be-processed sub-image data, and charge transfer low-efficiency removal processing is performed on each to-be-processed sub-image data through a pre-trained charge transfer low-efficiency removal model. The problem of low charge transfer efficiency caused by semiconductor material defects or impurities can be effectively solved. According to the method, false signals caused by low charge transfer efficiency of the CCD image can be effectively removed, so that the pixel value is accurately corrected, and the overall quality of the image is improved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and particularly to a CCD image processing method, apparatus, and storage medium. Background Art

[0002] A Charge Coupled Device (CCD) is an optical detector whose surface consists of many tiny photosensitive units (pixels or picture elements), and these photosensitive units are arranged in a two-dimensional array. When light irradiates the surface of the CCD, each photosensitive unit generates a certain amount of charge according to the received light intensity. This process is based on the photoelectric effect, that is, the energy of photons is absorbed by the semiconductor material, exciting electron-hole pairs. The stronger the light, the more charge is generated. The charge generated by each photosensitive unit is temporarily stored inside the photosensitive unit until, under the action of periodic voltage pulses, each photosensitive unit transfers the charge it has accumulated to the adjacent next photosensitive unit one by one, and then the adjacent photosensitive unit transfers the charge accumulated in this unit to the next photosensitive unit (usually it can be row-by-row transfer or column-by-column transfer) until it is transferred to the register set at the edge of the CCD surface, and the register converts it into an electrical signal, and the intensity of these electrical signals corresponds to the light intensity received by each photosensitive unit. For example: the stronger the light received by a photosensitive unit, the more charge it generates, and the stronger the read electrical signal. Furthermore, through a computer or other processing devices, the electrical signal corresponding to each photosensitive unit can be converted into the brightness value of each pixel point that makes up the CCD image, thereby obtaining the CCD image.

[0003] However, during the CCD imaging process, the quality of the obtained CCD image may be poor due to the Charge Transfer Inefficiency (CTI) problem, that is, because defects or impurities in the semiconductor material also have the ability to capture charge, so that the charge accumulated by a photosensitive unit cannot all be transferred to the register, and these captured charges will cause the problem of charge transfer inefficiency.

[0004] Therefore, how to remove the influence of the charge transfer inefficiency problem existing in the CCD image to improve the quality of the CCD image is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a CCD image processing method, apparatus, and storage medium to partially solve the above problems existing in the prior art.

[0006] This specification adopts the following technical solutions: This specification provides a CCD image processing method, including: Obtain the to-be-processed image data collected by the charge-coupled device; Split the to-be-processed image data into to-be-processed sub-image data; wherein, each to-be-processed sub-image data is used to represent the pixel values of a row or a column of pixel points in the to-be-processed image data; For each to-be-processed sub-image data, input the to-be-processed sub-image data into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the to-be-processed sub-image data; Stitch the processed sub-image data to obtain the target image data.

[0007] This specification provides a CCD image processing device, including: An acquisition module, configured to obtain the to-be-processed image data collected by the charge-coupled device; A splitting module, configured to split the to-be-processed image data into to-be-processed sub-image data; wherein, each to-be-processed sub-image data is used to represent the pixel values of a row or a column of pixel points in the to-be-processed image data; A processing module, configured to, for each to-be-processed sub-image data, input the to-be-processed sub-image data into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the to-be-processed sub-image data; An output module, configured to stitch the processed sub-image data to obtain the target image data.

[0008] This specification provides a computer-readable storage medium, where the storage medium stores a computer program, and when the computer program is executed by a processor, the above CCD image processing method is implemented.

[0009] This specification provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the above CCD image processing method is implemented.

[0010] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: In the CCD image processing method provided in this specification, first, obtain the to-be-processed image data collected by the charge-coupled device, split the to-be-processed image data into to-be-processed sub-image data, wherein, each to-be-processed sub-image data is used to represent the pixel values of a row or a column of pixel points in the to-be-processed image data, for each to-be-processed sub-image data, input the to-be-processed sub-image data into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the to-be-processed sub-image data, and stitch the processed sub-image data to obtain the target image data.

[0011] As can be seen from the above method, the server can split the image data to be processed into sub-image data to be processed, and perform charge transfer inefficiency removal processing on each sub-image data to be processed through a pre-trained charge transfer inefficiency removal model, which can effectively remove the charge transfer inefficiency problem caused by semiconductor material defects or impurities. This method can effectively remove the pseudo-signals caused by charge transfer inefficiency in CCD images, thereby accurately correcting pixel values and improving the overall quality of the images. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings described herein are used to provide a further understanding of the present specification and form a part of the present specification. The schematic embodiments of the present specification and their descriptions are used to explain the present specification and do not constitute an improper limitation to the present specification. In the drawings: Figure 1 is a schematic flowchart of a CCD image processing method provided in the present specification; Figure 2 is a schematic diagram of the pixel value distribution of an astronomical image taken by the Hubble Telescope provided in the present specification; Figure 3 is a schematic diagram of the preprocessing process of the image data to be processed provided in the present specification; Figure 4 is a schematic diagram of the effect after unilateral variable scaling normalization processing provided in the present specification; Figure 5 is a schematic diagram of the effect after bilateral variable scaling normalization processing provided in the present specification; Figure 6 is a schematic diagram of the charge transfer inefficiency removal model provided in the present specification; Figure 7A is a histogram of the relative error distribution obtained based on the test set provided in the present specification; Figure 7B is a schematic diagram of the target image data provided in the present specification; Figure 8 is a schematic diagram of the comparison of the CCD image processing speed provided in the present specification; Figure 9 is a schematic diagram of a CCD image processing device provided in the present specification; Figure 10 is a schematic diagram of an electronic device for CCD image processing provided in the present specification. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] To make the objectives, technical solutions, and advantages of this specification clearer, the following will clearly and completely describe the technical solutions of this specification in combination with specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, rather than all the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this specification.

[0014] The following will, in combination with the drawings, elaborate on the technical solutions provided by each embodiment of this specification.

[0015] Charge Transfer Efficiency (CTE) refers to how much charge can be correctly transferred from one photosensitive unit to another during the CCD imaging process. The ideal charge transfer efficiency during the CCD imaging process is 100%. However, since defects or impurities in semiconductor materials also have the ability to capture charge, in actual application scenarios, the charge transfer efficiency often fails to reach the ideal state. And since during the CCD imaging process, it is necessary to determine the pixel value of the pixel point corresponding to this photosensitive unit in the finally obtained image based on the amount of charge accumulated in each photosensitive unit, therefore, under the influence of the charge transfer inefficiency problem, the accuracy of the finally obtained image is often relatively low.

[0016] Based on this, in fields such as remote sensing, astronomy, biology, and medicine, there are often strict requirements for CCD images. Therefore, in these fields, removing Charge Transfer Inefficiency (CTI) from CCD images (removing the negative impact of the charge transfer inefficiency problem on CCD imaging) is particularly important.

[0017] For example: In the application of astronomical images, since cosmic rays carry a large number of high-energy high-speed particles, they can damage the CCD and increase the number of defects in the semiconductor. Therefore, there is a serious CTI effect. And the fields that require the application of astronomical images have extremely high requirements for the performance of image sensors. Therefore, the CTI problem is particularly prominent in these fields.

[0018] Figure 1 The following is a schematic flowchart of a CCD image processing method provided in this specification, including the following steps: S101: Obtain the to-be-processed image data collected by the charge-coupled device.

[0019] In this specification, the execution entity for implementing the CCD image processing method can refer to a specified device such as a server installed in a data processing center or an image analysis platform, or it can also refer to a terminal device such as a desktop computer, a laptop computer, or a professional graphics workstation. For the sake of convenience in description, hereinafter, only taking the server as the execution entity as an example, the CCD image processing method provided in this specification will be described.

[0020] In this specification, the server can perform charge transfer inefficiency CTI removal processing on the CCD image data collected by the charge-coupled device through a pre-trained charge transfer inefficiency removal model to obtain the processed image data, and then can perform task execution based on the processed image data.

[0021] Prior to this, the server can first obtain the CCD image data collected by the charge-coupled device as the image data to be processed.

[0022] It should be noted that the above image data to be processed can be two-dimensional matrix data used to represent the pixel values of each pixel point constituting the CCD image.

[0023] S102: Split the image data to be processed into sub-image data to be processed; wherein, each sub-image data to be processed is used to represent the pixel values of a row or a column of pixel points in the image data to be processed.

[0024] In an actual application scenario, during the CCD imaging process, each photosensitive unit transfers the charge accumulated by itself to the adjacent next photosensitive unit in the form of row-by-row transfer or column-by-column transfer, and then the adjacent photosensitive unit transfers the charge accumulated by this unit to the next photosensitive unit until it is transferred to the register set at the edge of the CCD surface, and the register converts it into an electrical signal. Then, based on the electrical signal converted by the register, the pixel value corresponding to the photosensitive unit in the generated CCD image can be determined.

[0025] Therefore, in the image data to be processed, the pixel values of each row or each column of pixels are affected by charge transfer inefficiency differently. Therefore, after the server obtains the image data to be processed, it can split the image data to be processed into sub-image data to be processed, and then can process each sub-image data to be processed to obtain the processed sub-image data.

[0026] Wherein, each sub-image data to be processed is used to represent the pixel values of a row or a column of pixel points in the image data to be processed.

[0027] It should be noted that for scientific image data, there are often problems such as a large and unbalanced numerical distribution range, specifically as Figure 2 shown.

[0028] Figure 2 It is a schematic diagram of the pixel value distribution of the astronomical image taken by the Hubble Telescope provided in this specification.

[0029] In Figure 2 , the abscissa is different pixel values, and the ordinate is the number of pixel points in each pixel point included in the image data whose pixel values are the pixel values corresponding to the abscissa. Combining Figure 2 It can be seen that in the astronomical image taken by the Hubble Telescope, a large number of background pixel values are concentrated in a very small range, and a small number of values are distributed in a very large numerical range. This makes it extremely uneven in numerical distribution after using traditional normalization methods such as zscore for high-precision processing of pixel values.

[0030] Based on this, in this specification, the server can perform variable scaling normalization processing on each sub-image data to be processed according to the predicted variable scaling normalization strategy, and obtain each preprocessed sub-image data to be processed, specifically as Figure 3 shown.

[0031] Figure 3 It is a schematic diagram of the preprocessing process of the image data to be processed provided in this specification.

[0032] Combining Figure 3 It can be seen that after dividing the data to be processed into each sub-image data to be processed, the server can also perform variable scaling normalization processing on each sub-image data to be processed according to the predicted variable scaling normalization strategy, and obtain each preprocessed sub-image data to be processed, so as to perform image processing on each preprocessed sub-image data to be processed, thereby obtaining the final processed sub-image data.

[0033] Among them, the above variable scaling normalization strategy includes: one-sided variable scaling normalization strategy or two-sided variable scaling normalization strategy. The above one-sided variable scaling normalization strategy and two-sided variable scaling normalization strategy can be selected according to the pixel value distribution of each pixel point included in the image data to be processed. For example: if the influence of negative numerical values (such as pixel values of overexposed and underexposed pixel points) in the image data to be processed does not need to be considered, the one-sided variable scaling normalization strategy can be selected; otherwise, the two-sided variable scaling normalization strategy can be selected.

[0034] Specifically, if the server uses the one-sided variable scaling normalization strategy to perform variable scaling normalization processing on each sub-image data to be processed to obtain each preprocessed sub-image data to be processed. It can specifically include the following steps: Step 1: The server can clip the pixel value of each pixel point according to the effective range vmin, vmax of the pixel values of the pixel points included in the image data to be processed, that is, make the value less than vmin equal to vmin and the value greater than vmax equal to vmax.

[0035] Step 2: Perform one-sided variable scaling normalization on the pixel value of each pixel point included in the sub-image data to be processed to obtain the pre-processed sub-image data to be processed. Specifically, the following formula can be referred to:

[0036] In the above formula, is the pixel value of the pixel point before the variable scaling normalization process, is the pixel value of the pixel point before the variable scaling normalization process. The values of a and add are used to adjust the shape and skip the rapid change interval of the logarithmic function, which can be set according to actual needs. As Figure 4 shown.

[0037] Figure 4 Figure 18 is a schematic diagram of the effect after the one-sided variable scaling normalization provided in this specification.

[0038] In Figure 4 Figure 23, part a of the figure is a schematic diagram of the correspondence between the serial number and the pixel value of the pixel point before the one-sided variable scaling normalization process. Part b of the figure is a schematic diagram of the correspondence between the serial number and the pixel value of the pixel point after the one-sided variable scaling normalization process. Among them, the abscissa is the serial number of the pixel point in the sub-image to be processed where the pixel point is located, and the ordinate is the pixel value of the pixel point.

[0039] Combining part a and part b of the above figure, it can be seen that the server can make the numerical interval with dense distribution become sparse and the sparse numerical interval become dense by performing one-sided variable scaling normalization on the sub-image data to be processed.

[0040] Furthermore, if the server uses a two-sided variable scaling normalization strategy to perform variable scaling normalization on each sub-image data to be processed to obtain the pre-processed sub-image data to be processed. Specifically, the following steps can be included: Step 1: The server can clip the pixel value of each pixel point according to the effective range vmin, vmax of the pixel values of the pixel points included in the image data to be processed, that is, make the value less than vmin equal to vmin and the value greater than vmax equal to vmax.

[0041] Step 2: Set the intermediate value, that is, the value med corresponding to the position where the numerical distribution is densest. For each pixel point in the sub-image data to be processed, adjust the pixel value of this pixel point according to the intermediate value med. Specifically, the following formula can be referred to: If the pixel value x of this pixel point is greater than med:

[0042] If the pixel value x of this pixel point is less than med:

[0043] Step 3: Perform bilateral variable scaling normalization processing on the pixel values of each pixel point included in the sub-image data to be processed, so as to obtain the preprocessed sub-image data to be processed. Specifically, the following formula can be referred to:

[0044]

[0045]

[0046] In the above formula, is the pixel value of the pixel point before the variable scaling normalization processing, is the pixel value of the pixel point before the variable scaling normalization processing. The a and add values are used to adjust the shape and skip the rapid change interval of the logarithmic function, and can be set according to actual needs. As Figure 5 shown.

[0047] Figure 5 is the schematic diagram of the effect after the bilateral variable scaling normalization processing provided in this specification.

[0048] In Figure 5 c part of the image is the schematic diagram of the corresponding relationship between the serial number and the pixel value of the pixel point before the bilateral variable scaling normalization processing. d part of the image is the schematic diagram of the corresponding relationship between the serial number and the pixel value of the pixel point after the bilateral variable scaling normalization processing. Among them, the abscissa is the serial number of the pixel point in the sub-image to be processed where this pixel point is located, and the ordinate is the pixel value of this pixel point.

[0049] Combining the above c part of the image and d part of the image, it can be seen that the server can perform bilateral variable scaling normalization processing on the sub-image data to be processed, so that the numerical interval with dense distribution becomes sparse, while the sparse numerical interval becomes dense.

[0050] S103: For each sub-image data to be processed, input this sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to this sub-image data to be processed.

[0051] Further, for each preprocessed sub-image data to be processed, the server can input the preprocessed sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the preprocessed sub-image data to be processed.

[0052] Of course, the server can also directly input each sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed.

[0053] In an actual application scenario, to improve the processing efficiency of CCD images, the server can also divide the sub-image data to be processed into each batch processing data set. Furthermore, for each batch processing data set, each sub-image data to be processed included in the batch processing data set can be input into a pre-trained charge transfer inefficiency removal model, so that the charge transfer inefficiency removal model performs parallel processing on each sub-image data to be processed included in the batch processing data set to obtain the processed sub-image data corresponding to each sub-image data to be processed included in the batch processing data set.

[0054] Among them, the size of the above batch processing data set can be set according to actual needs. For example: if the size of each batch processing data set is batch_size, the server can divide w sub-image data to be processed into W / / batch_size batch processing data sets, that is, W / / batch_size matrix data with a shape of H x batch_size x 1.

[0055] It should be noted that the above charge transfer inefficiency removal model can be deployed to the server for CCD image data only after being trained. Among them, the method for training the above charge transfer inefficiency removal model can be to obtain sample image data, split the sample image data into each sample sub-image data, and for each sample sub-image data, input the sample sub-image data into the charge transfer inefficiency removal model to be trained to obtain the processed sample sub-image data corresponding to the sample sub-image data. Furthermore, the processed sample sub-image data can be spliced to obtain the processed sample image data, and the target loss value can be determined according to the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and the charge transfer inefficiency removal model to be trained can be trained with minimizing the target loss value as the optimization goal to obtain the trained charge transfer inefficiency removal model.

[0056] Among them, the greater the deviation between the above processed sample image data and the ground truth image data corresponding to the sample image data, the greater the determined target loss value.

[0057] In an actual application scenario, the server can also determine a target loss value based on the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and use minimizing the target loss value as the optimization objective to train the charge transfer inefficiency removal model to be trained, obtaining an alternative charge transfer inefficiency removal model. The above steps are repeated at least once to obtain each alternative charge transfer inefficiency removal model.

[0058] Among them, different alternative charge transfer inefficiency removal models are obtained by training the charge transfer inefficiency removal model to be trained using different sample image data.

[0059] Furthermore, for each alternative charge transfer inefficiency removal model, the preset verification image data can be input into the alternative charge transfer inefficiency removal model, and based on the deviation between the processed verification image data output by the alternative charge transfer inefficiency removal model for the verification image data and the ground truth verification image data corresponding to the verification image data, the evaluation loss value of the alternative charge transfer inefficiency removal model can be determined.

[0060] On this basis, the server can select the alternative charge transfer inefficiency removal model with the smallest evaluation loss value from each alternative charge transfer inefficiency removal model as the trained charge transfer inefficiency removal model.

[0061] It should be noted that in an actual application scenario, the sub-image data to be processed are often processed in parallel in the form of a batch processing dataset. At this time, the charge transfer inefficiency removal model for each sub-image data to be processed can be interfered by the features of other sub-image data to be processed, resulting in a decrease in the accuracy of the processed sub-image data.

[0062] Therefore, the above charge transfer inefficiency removal model can also be composed of a main model and a conditional fusion model. At this time, for each sub-image data to be processed, the server can input the identification information (such as: row ID or column ID) and timestamp of the sub-image data to be processed among the sub-image data to be processed into the conditional fusion model to obtain a modulation parameter, and can input the sub-image data to be processed and the modulation parameter into the main model of the pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed.

[0063] As can be seen from the above content, the server can generate modulation parameters according to the identification information and timestamp of each sub-image data to be processed by introducing a conditional fusion model. These parameters can reflect the specific characteristics of each sub-image data to be processed (such as the defects existing in the semiconductor material where each photosensitive unit corresponding to the sub-image data to be processed is located), so as to provide personalized processing strategies for each sub-image data to be processed, and solve the problems of feature difference retention and interference that may exist between different sub-image data to be processed in batch processing. Specifically, as Figure 6 shown.

[0064] Figure 6 is a schematic diagram of the charge transfer inefficiency removal model provided in this specification.

[0065] Combined with Figure 6 it can be seen that the main model in the charge transfer inefficiency removal model includes: at least one intermediate processing layer, where the intermediate processing layer includes: an intermediate convolutional layer and an activation layer.

[0066] In an actual application scenario, the above main model may further include: a first convolutional layer, at least one intermediate processing layer, and a second convolutional layer. Among them, the intermediate processing layer includes: an intermediate convolutional layer, a batch normalization layer, and an activation layer. If the hidden layer variable dimension of the above main model is D_hidden, then the number of input channels of the first convolutional layer is 1, and the number of output channels is D_hidden. The number of input channels of the second convolutional layer is D_hidden, and the output is 1. The input and output dimensions of each intermediate convolutional layer are both D_hidden.

[0067] Specifically, the role of the first convolutional layer is to map the input single-channel image data (i.e., the sub-image data to be processed) to a higher-dimensional feature space, so as to extract the initial features of the sub-image data to be processed. The number of its convolutional kernels is D_hidden, and each convolutional kernel is responsible for extracting a specific feature of the input image to output D_hidden feature maps. These feature maps can capture local textures, edges, etc. in the sub-image data to be processed, providing a rich feature basis for subsequent processing.

[0068] The intermediate processing layer is then used to further process and optimize the D_hidden feature maps.

[0069] Among them, the intermediate convolutional layer continues to extract local features. Through the multi-convolutional kernel design, it can extract more complex features from the input D_hidden feature maps.

[0070] In the above content, the calculation process of the i-th output feature map can refer to the following formula:

[0071] In the above formula, I is the input feature map, is the i-th feature map obtained by calculation, is the i-th convolution kernel, is the i-th bias term.

[0072] Furthermore, the above batch normalization layer is used to normalize the output of the intermediate convolution layer, eliminate the internal covariate shift, accelerate the training process of the model and improve the stability of the model.

[0073] The activation layer (such as ReLU) is used to introduce non-linear factors, enabling the model to learn more complex feature mapping relationships.

[0074] The role of the second convolution layer is to map the D_hidden-dimensional feature map output by the intermediate processing layer back to a single-channel output, generating the final processed sub-image data. The number of convolution kernels in this layer is 1, and its purpose is to fuse the extracted complex features to obtain the final processed sub-image data.

[0075] During this process, the server can input the to-be-processed sub-image data and the modulation parameters into the main model of the pre-trained charge transfer inefficiency removal model, so that each intermediate processing layer of the main model linearly transforms and modulates the intermediate feature map output by the intermediate convolution layer in this intermediate processing layer according to the modulation parameters, obtains the modulated intermediate feature map, and inputs it into the activation layer to obtain the processed sub-image data corresponding to the to-be-processed sub-image data.

[0076] Specifically, the conditional fusion model can input the identification information and timestamp of the to-be-processed sub-image data in each to-be-processed sub-image data into the conditional encoder of the conditional fusion model to obtain the modulation parameters beta, alpha output by the conditional fusion model. The dimensions of the modulation parameters beta, alpha here are both N*D_hidden. Furthermore, during the processing of each intermediate processing layer of the main model, before the intermediate feature map output by the batch normalization layer is input into the activation layer, the intermediate feature map output by the batch normalization layer is linearly transformed and modulated. Among them, if the j-th dimension of the intermediate feature map output by the batch normalization layer before the convolution layer of the i-th intermediate processing layer is F_i_j, then the j-th dimension of the modulated intermediate feature map after modulation is FM_i_j. The specific formula can be referred to as follows:

[0077] Furthermore, the server can also splice the to-be-processed sub-image data input into the main model into the feature map output by the above second convolution layer in the form of a residual connection as the target feature map, so as to force the charge transfer inefficiency removal model to learn the differential part of the output data and reduce the complexity of learning of the charge transfer inefficiency removal model.

[0078] Further, the server can perform inverse scaling normalization processing on the target feature map to obtain the processed sub-image data corresponding to the sub-image data to be processed.

[0079] It should be noted that the operation of the inverse scaling normalization processing here is the same as that of the above-mentioned variable scaling normalization processing, except that the input and output are reversed, and this specification will not elaborate here.

[0080] S104: Concatenate the processed sub-image data to obtain the target image data.

[0081] In this specification, after the server obtains the processed sub-image data, it can concatenate the processed sub-image data to obtain the target image data, and can perform tasks according to the target image data.

[0082] Among them, the above tasks can be determined according to the actual application scenario. For example, in the field of remote sensing, the target image data can be used to perform tasks such as land use classification, environmental monitoring, and assessment of crop growth conditions. By removing the influence of charge transfer inefficiency (CTI), the processed image can more accurately reflect the surface features and changes. For example, high-resolution remote sensing images can be used to monitor deforestation, urban expansion, or regional changes after natural disasters, providing more reliable data support for environmental protection and resource management.

[0083] Another example: In the application of astronomical images, the target image data can be used for tasks such as galaxy classification, star observation, and cosmic background radiation research. Since the damage of cosmic rays to the CCD will cause a serious CTI effect, the image after removing CTI can more clearly display the details and structures of celestial bodies. For example, in deep space observation, the processed image can be used to study the morphology and evolution of distant galaxies, or to discover new celestial bodies such as asteroids and comets.

[0084] As can be seen from the above method, the server can split the image data to be processed into each sub-image data to be processed, and perform charge transfer inefficiency removal processing on each sub-image data to be processed through a pre-trained charge transfer inefficiency removal model, which can effectively remove the charge transfer inefficiency problem caused by semiconductor material defects or impurities. This method can remove the pseudo-signals brought by charge transfer inefficiency in CCD images, thereby accurately correcting the pixel values to improve the overall quality of the image. Specifically as Figure 7A 、 Figure 7B shown.

[0085] Figure 7A This is the relative error distribution histogram obtained based on the test set in this specification.

[0086] FromFigure 7A It can be seen from the relative error distribution histogram generated by processing 100 4096*4096 test sets that the relative error value of the image data processed by the above CCD image processing method is closer to 0. This shows that the above method can effectively remove the influence of charge transfer inefficiency (CTI), so that the difference between the processed image and the ideal image is significantly reduced. The closer the relative error is to 0, the better the image restoration effect is and the more significant the image quality improvement is.

[0087] Figure 7B This is a schematic diagram of the target image data provided in this specification.

[0088] exist Figure 7B In the figure, the image data on the left is the image data to be processed, the image data in the middle is the target image data, and the image data on the right is the true value image data corresponding to the image data to be processed. Figure 7B It can be seen that the above CCD image processing method can remove the false signal caused by the inefficient charge transfer of the CCD image, thereby accurately correcting the pixel value.

[0089] In addition, the processing speed of CCD image processing can be effectively improved by processing the image data to obtain the target image data through the above method. Figure 8 shown.

[0090] from Figure 8 It can be seen that when a graphics card with a graphics card model of Rt*2080Ti is used to execute the CCD image processing method provided in this manual, the time consumed for CCD image processing is 1.19 seconds per image. When a graphics card with a graphics card model of V100 is used to execute the CCD image processing method provided in this manual, the time consumed for CCD image processing is 0.8 seconds per image. In the prior art, the time consumed for high-precision CCD image processing using the SimpleCTI algorithm is 1990.375 seconds per image, and the time consumed for low-precision CCD image processing is 19.647 seconds per image. The time consumed for high-precision CCD image processing using the Arctic algorithm is 10161.74 seconds per image, and the time consumed for low-precision CCD image processing is 271.078 seconds per image. It can be seen that the above-mentioned CCD image processing method can not only effectively remove the pseudo signal caused by the inefficient charge transfer of the CCD image, thereby accurately correcting the pixel value. It can also effectively improve the speed of CCD image processing.

[0091] The above are one or more CCD image processing methods implemented in this specification. Based on the same idea, this specification also provides a corresponding CCD image processing device, such as Figure 9 shown.

[0092] Figure 9 Schematic diagram of a CCD image processing device provided in this specification, including: An acquisition module 901, configured to acquire to-be-processed image data collected by a charge-coupled device; A splitting module 902, configured to split the to-be-processed image data into to-be-processed sub-image data; wherein each to-be-processed sub-image data is used to represent pixel values of a row or a column of pixel points in the to-be-processed image data; A processing module 903, configured to input each to-be-processed sub-image data into a pre-trained charge transfer inefficiency removal model for each to-be-processed sub-image data, and obtain processed sub-image data corresponding to the to-be-processed sub-image data; An output module 904, configured to splice the processed sub-image data to obtain target image data.

[0093] Optionally, the processing module 903 is specifically configured to divide the to-be-processed sub-image data into each batch processing data set; for each batch processing data set, input each to-be-processed sub-image data included in the batch processing data set into a pre-trained charge transfer inefficiency removal model, so that the charge transfer inefficiency removal model performs parallel processing on each to-be-processed sub-image data included in the batch processing data set, and obtain processed sub-image data corresponding to each to-be-processed sub-image data included in the batch processing data set.

[0094] Optionally, the processing module 903 is specifically configured to perform variable scaling normalization processing on each to-be-processed sub-image data according to a predicted variable scaling normalization strategy to obtain each pre-processed to-be-processed sub-image data; the variable scaling normalization strategy includes: one-sided variable scaling normalization strategy, bilateral variable scaling normalization strategy; for each pre-processed to-be-processed sub-image data, input the pre-processed to-be-processed sub-image data into a pre-trained charge transfer inefficiency removal model, and obtain processed sub-image data corresponding to the pre-processed to-be-processed sub-image data.

[0095] Optionally, the device further includes: a training module 905; The training module 905 is specifically configured to obtain sample image data, and split the sample image data into respective sample sub-image data; for each sample sub-image data, input the sample sub-image data into the charge transfer inefficiency removal model to be trained, and obtain the processed sample sub-image data corresponding to the sample sub-image data; splice the processed sample sub-image data to obtain processed sample image data; determine a target loss value according to the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and use minimizing the target loss value as an optimization objective to train the charge transfer inefficiency removal model to be trained, so as to obtain a trained charge transfer inefficiency removal model.

[0096] Optionally, the training module 905 is specifically configured to determine a target loss value according to the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and use minimizing the target loss value as an optimization objective to train the charge transfer inefficiency removal model to be trained, so as to obtain an alternative charge transfer inefficiency removal model; repeat the above steps at least once to obtain respective alternative charge transfer inefficiency removal models; wherein, different alternative charge transfer inefficiency removal models are obtained by training the charge transfer inefficiency removal model to be trained using different sample image data; input preset verification image data into each alternative charge transfer inefficiency removal model to obtain an evaluation loss value of each alternative charge transfer inefficiency removal model; select a trained charge transfer inefficiency removal model from the alternative charge transfer inefficiency removal models according to the evaluation loss value.

[0097] Optionally, the charge transfer inefficiency removal model includes: a main model, a conditional fusion model; The processing module 903 is specifically configured to, for each sub-image data to be processed, input the identification information and time stamp of the sub-image data to be processed among the sub-image data to be processed into the conditional fusion model to obtain a modulation parameter; input the sub-image data to be processed and the modulation parameter into the main model of the pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed.

[0098] Optionally, the main model includes: at least one intermediate processing layer; the intermediate processing layer includes: an intermediate convolutional layer and an activation layer; Specifically, the processing module 903 is configured to input the sub-image data to be processed and the modulation parameters into the main model of the pre-trained charge transfer inefficiency removal model, so that each intermediate processing layer of the main model performs linear transformation modulation on the intermediate feature map output by the intermediate convolutional layer in this intermediate processing layer according to the modulation parameters, obtains a modulated intermediate feature map, and inputs it into the activation layer to obtain the processed sub-image data corresponding to the sub-image data to be processed.

[0099] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above Figure 1 provided CCD image processing method.

[0100] This specification also provides Figure 10 a schematic structural diagram of an electronic device corresponding to Figure 1 as shown. At the hardware level, as Figure 10 described, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 described CCD image processing method. Of course, in addition to the software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0101] Those skilled in the art should understand that the embodiments of this specification can be provided as a method, a system, or a computer program product. Therefore, this specification can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This specification can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification can also be practiced in a distributed computing environment where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0103] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the corresponding description in the method embodiment.

[0104] The above description is only for the embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various changes and modifications can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this specification.

Claims

1. A CCD image processing method, characterized in that, Including: Obtain the image data to be processed collected by the charge-coupled device; Split the image data to be processed into sub-image data to be processed; wherein, each sub-image data to be processed is used to represent the pixel values of a row or a column of pixel points in the image data to be processed; For each sub-image data to be processed, input the sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed; Stitch the processed sub-image data to obtain the target image data.

2. The method according to claim 1, characterized in that For each sub-image data to be processed, inputting the sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed specifically includes: Divide the sub-image data to be processed into batch processing data sets; For each batch processing data set, input each sub-image data to be processed included in the batch processing data set into a pre-trained charge transfer inefficiency removal model, so that the charge transfer inefficiency removal model performs parallel processing on each sub-image data to be processed included in the batch processing data set, and obtain the processed sub-image data corresponding to each sub-image data to be processed included in the batch processing data set.

3. The method according to claim 1, characterized in that Before inputting each sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed, the method further includes: Perform variable scaling normalization processing on each sub-image data to be processed according to the predicted variable scaling normalization strategy to obtain the pre-processed sub-image data to be processed; the variable scaling normalization strategy includes: one of the unilateral variable scaling normalization strategy and the bilateral variable scaling normalization strategy; For each sub-image data to be processed, inputting the sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed specifically includes: For each pre-processed sub-image data to be processed, input the pre-processed sub-image data to be processed into a pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the pre-processed sub-image data to be processed.

4. The method according to claim 1, wherein The method further includes: training a charge transfer inefficiency removal model, specifically including: Obtain sample image data and split the sample image data into sample sub-image data; For each sample sub-image data, input the sample sub-image data into the charge transfer inefficiency removal model to be trained to obtain the processed sample sub-image data corresponding to the sample sub-image data; Stitch the processed sample sub-image data to obtain the processed sample image data; Determine the target loss value according to the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and train the charge transfer inefficiency removal model to be trained with minimizing the target loss value as the optimization goal to obtain the trained charge transfer inefficiency removal model.

5. The method according to claim 4, wherein Determine a target loss value based on the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and use minimizing the target loss value as the optimization objective to train the charge transfer inefficiency removal model to be trained, obtaining a trained charge transfer inefficiency removal model, specifically including: Determine a target loss value based on the deviation between the processed sample image data and the ground truth image data corresponding to the sample image data, and use minimizing the target loss value as the optimization objective to train the charge transfer inefficiency removal model to be trained, obtaining an alternative charge transfer inefficiency removal model; Repeat the above steps at least once to obtain each alternative charge transfer inefficiency removal model; wherein, different alternative charge transfer inefficiency removal models are obtained by training the charge transfer inefficiency removal model to be trained using different sample image data; Input the preset verification image data into each alternative charge transfer inefficiency removal model to obtain the evaluation loss value of each alternative charge transfer inefficiency removal model; Select the trained charge transfer inefficiency removal model from each alternative charge transfer inefficiency removal model according to the evaluation loss value.

6. The method according to claim 1, characterized in that, The charge transfer inefficiency removal model includes: a main model, a conditional fusion model; For each sub-image data to be processed, input the sub-image data to be processed into the pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed, specifically including: For each sub-image data to be processed, input the identification information and timestamp of the sub-image data to be processed among the sub-image data to be processed into the conditional fusion model to obtain a modulation parameter; Input the sub-image data to be processed and the modulation parameter into the main model of the pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed.

7. The method according to claim 6, wherein The main model includes: at least one intermediate processing layer; the intermediate processing layer includes: an intermediate convolutional layer and an activation layer; Input the sub-image data to be processed and the modulation parameter into the main model of the pre-trained charge transfer inefficiency removal model to obtain the processed sub-image data corresponding to the sub-image data to be processed, specifically including: Input the sub-image data to be processed and the modulation parameter into the main model of the pre-trained charge transfer inefficiency removal model, so that each intermediate processing layer of the main model linearly transforms and modulates the intermediate feature map output by the intermediate convolutional layer in the intermediate processing layer according to the modulation parameter to obtain a modulated intermediate feature map, and input it into the activation layer to obtain the processed sub-image data corresponding to the sub-image data to be processed.

8. A CCD image processing device, characterized in that, Including: An acquisition module, configured to acquire the image data to be processed collected by the charge coupled device; A splitting module, configured to split the image data to be processed into each sub-image data to be processed; wherein, each sub-image data to be processed is used to represent the pixel values of a row or a column of pixel points in the image data to be processed. A processing module, configured to input the sub-image data to be processed into a pre-trained charge transfer inefficiency removal model for each sub-image data to be processed, so as to obtain the processed sub-image data corresponding to the sub-image data to be processed; An output module, configured to splice the processed sub-image data to obtain target image data.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 above is implemented.

10. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the method described in any one of claims 1 to 7 above is implemented.

Images