Data processing method and system based on image acquisition equipment

Abstract

The invention discloses a data processing method and system based on image acquisition equipment, and relates to the technical field of computer vision, and the method comprises the steps: constructing CNN-VAE to generate a low-light image of a multi-modal image, employing an inverse attention module and a GAN model of a double-discriminator structure, and combining parallel expansion and expansion convolution to expand a training data set; images in the training data set are enhanced through double-branch U-Net, image details are enhanced through local variance adaptive Gaussian filtering, pixel gradients are classified through a support vector machine, and edge detection is carried out; resNet50 is adopted to extract features, and a candidate frame is accurately positioned in combination with RPN and non-maximum suppression. Through CNN-VAE and an anti-attention GAN model, training data is expanded, double-branch U-Net and local variance adaptive Gaussian filtering are utilized to enhance image details, edge detection is optimized through a support vector machine, ResNet50 feature extraction is adopted, an RPN network and non-maximum suppression are combined to position an object candidate frame, and high-precision object detection and recognition are realized. And the precision and robustness of image processing are obviously improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a data processing method and system based on an image acquisition device. Background Art

[0002] With the widespread application of image acquisition equipment, image processing technology has been greatly developed in many fields, especially in the fields of medical imaging, autonomous driving, security monitoring and remote sensing image analysis. The accurate processing and analysis of image data has become the core technology. In recent years, image processing methods based on deep learning have gradually become dominant, especially the widespread application of technologies such as convolutional neural networks (CNN) and generative adversarial networks (GAN), which have made significant progress in image classification, detection, enhancement and other tasks. CNN, as a powerful feature extraction tool, has been widely used in image recognition and segmentation tasks, while GAN has shown great potential in image generation and enhancement with its generation ability. At the same time, variational autoencoders (VAE) have been increasingly used in the field of image processing due to their advantages in generating models and data reconstruction. The combination of these technologies provides a more flexible solution for multimodal image processing.

[0003] However, existing image processing technologies still face many challenges in dealing with multimodal image processing. Multimodal images usually come from different sensor devices, such as RGB images and infrared images, and the lighting conditions, resolution and detail performance of the two are quite different. How to effectively fuse these images and ensure the accuracy of processing is still a difficult problem. Existing image enhancement methods, such as enhancement methods based on simple convolution operations, are often difficult to fully extract and retain the detailed information of the image, especially in low-light and shadow areas, where details are often lost. Although generative adversarial networks (GANs) can effectively enhance image quality, the images they generate still have certain quality problems under complex conditions such as uneven lighting and noise, especially in terms of edge detection and object positioning. Although the existing object detection methods based on deep learning have achieved good results in many fields, in the joint processing and analysis of multimodal data, existing technologies often fail to effectively combine the features of different modalities, resulting in limited accuracy and robustness of object recognition. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a data processing method and system based on an image acquisition device to solve the problems of insufficient image fusion, loss of details and inaccurate object detection when processing multimodal images in existing image processing technologies; In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a data processing method based on an image acquisition device, comprising: Acquire multimodal images from an image acquisition device and perform preprocessing; The multimodal image includes an RGB image and an infrared image; Integrate multimodal images into training datasets, build CNN-VAE to generate low-light images of multimodal images, use the GAN model with anti-attention module and dual discriminator structure, and combine parallel expansion and dilated convolution to expand the training dataset; The images in the training dataset are enhanced by using a dual-branch U-Net, the image details are enhanced by using a local variance adaptive Gaussian filter, and the pixel gradients are classified and edge detection is performed by a support vector machine. ResNet50 is used to extract features, and RPN and non-maximum suppression are combined to accurately locate candidate boxes. Object detection and recognition are performed through classification regression and visualized. All data is stored and managed in the database.

[0006] As a preferred solution of the data processing method based on the image acquisition device described in the present invention, wherein: the construction of CNN-VAE to generate low-light images of multimodal images, the use of the GAN model with an anti-attention module and a dual discriminator structure, combined with parallel expansion of the dilated convolution to expand the training data set includes: Use random rotation to simulate different directions of the preprocessed image, adjust the brightness of the multimodal image through histogram equalization and integrate it into the training data set, map the training data set to the latent space through the variational autoencoder, and sample the latent space data from the latent space , by maximizing the variational lower bound, the variational autoencoder learns the distribution of the latent space and generates low-light images by minimizing the loss function, and the generated low-light images are used as new samples to expand the training dataset; Use the generative adversarial network to expand the training dataset, define the anti-attention module, build a dual discriminator structure to extract the global and local features of the image and add them to each layer in the generator, use global average pooling to encode the spatial information of the multimodal images in the training dataset, perform nonlinear transformation through the Sigmoid activation function, generate the anti-attention map and suppress unnecessary areas by performing the reverse operation to generate the image; A parallel extended convolution module is integrated into the generative adversarial network. By using convolutions with different dilation rates in parallel, the receptive field of the generative adversarial network is expanded and image enhancement is performed, and the final enhanced data set is output.

[0007] As a preferred solution of the data processing method based on the image acquisition device of the present invention, wherein: the construction of the dual discriminator structure to extract the global features and local features of the image includes: The anti-attention module extracts the global and local features of the image, adjusts the weights of the feature channels according to the importance of the global and local features, uses PatchGAN as the local discriminator, and combines it with the global discriminator for training. The real image and the generated image are used to train the discriminator, calculate the discriminator loss, and use the Adam optimizer to train the model until convergence.

[0008] As a preferred solution of the data processing method based on the image acquisition device of the present invention, wherein: the image enhancement in the training data set by the dual-branch U-Net includes: The U-Net architecture is used as the joint learning framework and the anti-attention module is integrated for each convolutional layer. The RGB images and infrared images in the final enhanced dataset are respectively input into two parallel U-Net branches to generate enhanced images. The local features of the image are retained through skip connections. The parameters of U-Net are updated using the joint loss function. The optimizer is defined, the gradient is calculated and updated, and the U-Net model is optimized to obtain the final enhanced image.

[0009] As a preferred solution of the data processing method based on the image acquisition device of the present invention, the method of enhancing image details by using local variance adaptive Gaussian filtering and classifying pixel gradients and performing edge detection by using a support vector machine includes: The enhanced image is converted into a grayscale image, and the grayscale image is divided into fixed-size blocks to obtain multiple local sub-regions. The local grayscale variance value in each local sub-region is calculated and the grayscale variance values ​​of all local sub-regions are used as a one-dimensional distribution. The optimal grayscale variance threshold is determined by calculating the grayscale histogram of the grayscale image and the Otsu algorithm formula. Determine the unique local Gaussian smoothing window size for each sub-region according to the optimal grayscale variance threshold, perform deterministic Gaussian filtering on the grayscale value of each pixel position in each corresponding region of the original image, replace the original grayscale value with the filtered grayscale value, and deterministically splice each sub-region back to the original size of the complete image according to the original image position after filtering. Calculate the gradient value of each pixel in the image, calculate the gradient amplitude and direction through the Canny edge detection algorithm, and perform edge detection on each pixel; Select the support vector machine algorithm, use the RBF kernel as the kernel function, extract the features of each pixel from the enhanced image as the input features of the training data set, mark the edge detection of each pixel in the input features, use the labeled training data set, train the model by minimizing the loss function of the support vector machine, and solve the optimal hyperplane and bias by using the gradient descent method; The features of each pixel of the image are input into the trained support vector machine model for classification, and the edge probability of each pixel in the image is adjusted based on the output of the support vector machine model.

[0010] As a preferred solution of the data processing method based on the image acquisition device described in the present invention, wherein: the use of ResNet50 to extract features, combining RPN and non-maximum suppression to accurately locate candidate frames, and performing object detection and recognition through classification regression and visual display includes: Use the pre-trained ResNet50 model as the feature extractor; Use global average pooling to pool the features of each channel to obtain a one-dimensional feature vector, use FasterR-CNN's Region Proposal Network to generate object candidate boxes, use the non-maximum suppression algorithm to select the best candidate box from multiple object candidate boxes, use the Softmax classifier to classify each candidate box, determine the category of each candidate box by setting a threshold, perform regression on each candidate box, predict the specific position and size of the object in the box, visualize the object detection and recognition results, and generate the final output.

[0011] As a preferred solution of the data processing method based on the image acquisition device of the present invention, wherein: storing all data into the database and managing it includes: Select a relational database to manage data and relationship analysis results, design the database table structure to store different types of data, set up regular backup tasks, back up all data in the database, manage the permissions of database users, and encrypt and store static data.

[0012] In a second aspect, the present invention provides a data processing system based on an image acquisition device, comprising: A data acquisition module, used for acquiring multimodal images from an image acquisition device and performing preprocessing; The model building module is used to integrate multimodal images into a training dataset, build a CNN-VAE to generate low-light images of multimodal images, and use a GAN model with an anti-attention module and a dual discriminator structure, combined with parallel expansion and dilated convolution to expand the training dataset; Image enhancement module, used to enhance images in the training dataset through a dual-branch U-Net; Edge detection module, used to enhance image details using local variance adaptive Gaussian filtering, classify pixel gradients and perform edge detection through support vector machine; Feature extraction module, which uses ResNet50 to extract features and combines RPN and non-maximum suppression to accurately locate candidate boxes; The target recognition module is used to detect and identify objects through classification and regression and to visualize them; The data storage module is used to store and manage all data in the database.

[0013] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the data processing method based on the image acquisition device as described in the first aspect of the present invention is implemented.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the data processing method based on the image acquisition device as described in the first aspect of the present invention.

[0015] The beneficial effects of the present invention are as follows: training data is expanded through CNN-VAE and anti-attention GAN models, image details are enhanced by using dual-branch U-Net and local variance adaptive Gaussian filtering, edge detection is optimized through support vector machine, ResNet50 feature extraction is adopted, and RPN network and non-maximum suppression are combined to locate object candidate frames, so as to achieve high-precision object detection and recognition, and significantly improve the accuracy and robustness of image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0017] Figure 1 The figure is a flow chart of the data processing method based on the image acquisition device in the present invention.

[0018] Figure 2 It is a schematic diagram of a data processing system based on an image acquisition device in the present invention.

[0019] Figure 3 This is a flow chart of expanding the training data set in the present invention.

[0020] Figure 4 The figure is a flow chart of the support vector machine classifying pixel gradients and performing edge detection in the present invention. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0024] Example, see Figures 1 to 4 , this embodiment provides a data processing method based on an image acquisition device, comprising the following steps: S1, collecting multimodal images of an image acquisition device and performing preprocessing; Specifically, according to the task requirements, image acquisition equipment (such as low-light photography equipment and infrared sensors) is used to synchronously acquire multimodal images of the target area in a low-light environment (such as at night or on a cloudy day), assuming that the target scene is remote sensing monitoring; The RGB image is a three-channel image of red, green and blue, the data of each channel is a matrix, and the value of each pixel represents the color information of RGB; The infrared image is a single-channel grayscale image, and the acquisition timestamps are aligned to ensure that the multimodal images correspond to the same area; The preprocessing includes spatial alignment and resolution standardization of multimodal images, upsampling infrared images to the resolution of RGB images, calculating new pixel values ​​using bilinear interpolation, applying non-local mean denoising algorithms to RGB images and infrared images respectively, suppressing noise during device acquisition, checking missing areas of multimodal images and filling them using a low-rank approximation method based on matrix completion, and normalizing pixel values ​​of multimodal images.

[0025] The preprocessing process makes the infrared image and the RGB image have the same resolution through spatial alignment and resolution standardization, ensuring the spatial consistency of the image, which is convenient for subsequent analysis and fusion. The non-local mean denoising algorithm is used to effectively suppress the noise that may be generated during the acquisition process and improve the clarity of the image, especially in low-light environments. The low-rank approximation method of matrix completion is used to fill the missing areas in the multimodal image to ensure the integrity of the data. Pixel value normalization eliminates the brightness difference under different image sources and acquisition conditions, ensuring the unified scale of the image, which helps to improve the effect of subsequent deep learning model training, thereby improving the accuracy and robustness of remote sensing monitoring tasks.

[0026] S2. Integrate multimodal images into training datasets, build CNN-VAE to generate low-light images of multimodal images, use the GAN model with anti-attention module and dual discriminator structure, and combine parallel expansion and dilated convolution to expand the training dataset; Specifically, we construct a CNN-VAE to generate low-light images of multimodal images, adopt a GAN model with an anti-attention module and a dual discriminator structure, and combine parallel expansion of the dilated convolution to expand the training data set, including: Random rotation is used to simulate different directions of the preprocessed images, and the brightness of the multimodal images is adjusted through histogram equalization and integrated into the training dataset; In order to increase the diversity of the training data set and improve the robustness of the model, the training data set is mapped to the latent space through a variational autoencoder (VAE). The structure of the variational autoencoder includes an encoder, a decoder, and a latent space. The encoder is designed as a 5-layer convolutional neural network (CNN) to output the mean and variance of the latent space (the dimension is set to 128). The decoder is designed as a 5-layer deconvolution network to sample the latent space to obtain the latent space data. , the formula is: , in, represents the mean of the latent space, represents the variance of the latent space, represents standard normally distributed random noise; By maximizing the variational lower bound (ELBO), the variational autoencoder learns the distribution of the latent space and generates low-light images by minimizing the loss function, providing more samples for the model and enhancing the generalization ability of the model. The loss function is: , in, represents the total loss function value of the variational autoencoder, is the output of the encoder, indicating that given an input image and latent space data The distribution of is obtained by variational inference. is the output of the decoder, representing the given latent space data Under the condition of The probability distribution of represents the expected value, that is, the expectation operator, Represents KL divergence, which measures the encoder distribution and the prior distribution The difference between is the expected value part, indicating that given the latent space data When , the input image is reconstructed The log-likelihood of Reconstruct the input image in the decoder By maximizing this expected value, the simulation is able to generate output images that are similar to the input images. Representing latent space data Prior distribution of, variational autoencoders usually use standard normal distribution as prior distribution; Use the generated low-light images as new samples to expand the training dataset; In order to further optimize the image quality, the generative adversarial network (GAN) is used to expand the training data set, define the anti-attention module (AAB), build a dual discriminator structure to extract the global and local features of the image and add them to each layer in the generator, use global average pooling to encode the spatial information of the multimodal images in the training data set, perform nonlinear transformation through the Sigmoid activation function, generate an anti-attention map and suppress unnecessary areas by performing an anti-operation (suppressing light pollution areas through anti-attention operations and emphasizing dark details of the image), and generate images; Integrate the parallel dilated convolution module in the generative adversarial network, expand the receptive field of the generative adversarial network and perform image enhancement by using convolutions with different dilation rates in parallel (such as dilation rates of 1, 3, and 5), capture more contextual information, and output the final enhanced data set (including original images, VAE enhanced images, and high-quality images generated by GAN). The formula is: ， in, represents the enhanced image, represents the image stitching operation, Represents the image The dilated convolution operation, Represents the dilation rate and can extract features from multiple scales.

[0027] Based on the above, the proposed scheme significantly enhances the diversity and image quality of the training data set by combining a variety of advanced technologies, thereby improving the generalization ability and robustness of the model. Random rotation and histogram equalization are used to preprocess and adjust the brightness of multimodal images to expand the diversity of the training data set. Then, the data is mapped to the latent space through the variational autoencoder (VAE) to generate low-light images. These low-light images are used to expand the data set and enhance the adaptability of the model to low-light conditions. Through the generative adversarial network (GAN) and the anti-attention module (AAB), the generated images are further optimized to emphasize the dark details and suppress unnecessary areas such as light pollution to generate clearer and more natural low-light images. The parallel expanded dilated convolution module is integrated to extract multi-scale contextual information through convolutions with different dilation rates, and enhance the receptive field and quality of the generated images. Through these steps, the generated enhanced images can not only improve the performance of the model under low-light conditions, but also help the model perform better under different environmental conditions through the combination of multimodal data, thereby enhancing the robustness and accuracy of the model.

[0028] Furthermore, the dual discriminator structure is constructed to extract the global and local features of the image, including: The anti-attention module extracts the global features of the image (i.e., the overall brightness, contrast, and other information of the image) and local features (i.e., the shadows or textures of the dark parts of the image), ensuring that the generative adversarial network not only focuses on the overall perception of the image, but also processes the dark details. The weights of the feature channels are adjusted according to the importance of the global and local features. In this process, the weights of the dark areas are particularly strengthened, so that the dark details can be fully restored during the image generation process. PatchGAN is used as a local discriminator and combined with a global discriminator for training. Real images and generated images are used to train the discriminator, and the discriminator loss is calculated. The model is trained to convergence using the Adam optimizer. The global discriminator is responsible for evaluating the authenticity of the entire generated image and outputs a binary classification result to indicate whether the image is real. Its structure is generally based on a fully convolutional network (FCN). Its goal is to evaluate the large-scale features of the image (such as texture, hue, etc.), while the local discriminator focuses on the local details of the image. It often uses smaller convolution kernels to process multiple local areas of the image (such as a 32×32 area). It can refine the details of the local area of ​​the image during the generation process, such as edges, textures, etc., to ensure that the generated image can also achieve realism at the detail level; By combining the global discriminator with the local discriminator, the dual discriminant structure can enhance the generator's attention to image details, making the generated image not only look real as a whole but also more refined in details. The detail processing capability of the local discriminator can avoid distortion of the generated image in certain local areas (such as shadows or highlights).

[0029] By constructing a dual discriminator structure and combining it with an anti-attention module, the performance of the generative adversarial network (GAN) in the image generation process can be significantly improved. The anti-attention module extracts global features (such as brightness, contrast, etc.) and local features (such as shadow or dark texture), and dynamically adjusts the weights of the feature channels according to the importance of these features, especially strengthening the weights of the dark area, thereby ensuring that the dark details are fully restored. PatchGAN is used as the local discriminator, combined with a global discriminator based on a fully convolutional network (FCN). The dual discriminator structure enables the generator to not only focus on the overall perception of the image (such as texture and tone), but also to finely process local details (such as edges and textures). This structure effectively improves the realism of the generated image at the detail level and avoids distortion in local areas, especially in shadows and highlights. Ultimately, the generated image can achieve high quality in both overall and details, improving the realism and visual effects of the image.

[0030] S3, enhance the images in the training data set through dual-branch U-Net, enhance image details using local variance adaptive Gaussian filtering, and classify pixel gradients and perform edge detection through support vector machine; Specifically, the image enhancement in the training data set by the dual-branch U-Net includes: Use the U-Net architecture as a joint learning framework and integrate an anti-attention module for each convolutional layer to enhance the dark and detailed areas of the image. With the help of the anti-attention module, U-Net can generate clearer images under low light and shadow conditions. Input the RGB image and infrared image in the final enhanced dataset into two parallel U-Net branches to generate enhanced images. Preserve the local features of the image through skip connections. Use the joint loss function to update the parameters of U-Net, define the optimizer, calculate the gradient and update it, optimize the U-Net model, and obtain the final enhanced image. The joint loss function includes exposure consistency loss, color consistency loss and spatial consistency loss. Exposure consistency loss ensures that the exposure of the image is within a reasonable range to avoid the image being too dark or too bright. The loss function calculates the difference between the brightness of the local area of ​​the image and the target exposure range. , the loss function is: , in, Represents the enhanced images in the training dataset No. The brightness value of the local area, Indicates the ideal exposure range. Usually we hope that the brightness of the image is within this range, indicating that the exposure is moderate. Represents the total number of local areas in the image; The color consistency loss is used to avoid color distortion of the generated image and ensure that the RGB channel colors of the image are similar. This loss measures the color difference between the red, green, and blue channels of the image. , the loss function is: , in, , , Respectively represent the average intensity values ​​of the red, green, and blue channels after image enhancement; The spatial consistency loss ensures the consistency of the generated image with the original image in spatial structure and avoids the loss of image details. The loss calculates the intensity difference of the local area of ​​the image. , which reflects the degree of preservation of the image structure; , in, and Represent the enhanced image and the original image in The average intensity value of a local area; Calculate the joint loss function , the formula is: , in, , , Represents the weight of the loss function.

[0031] By performing image enhancement on the training dataset through a dual-branch U-Net architecture, the quality of images in low light and shadow conditions can be effectively improved. The U-Net architecture combined with the anti-attention module enables the model to focus on the dark and detail areas in the image, thereby enhancing the clarity and brightness of the image. This method processes RGB images and infrared images in parallel and uses jump connections to retain the local features of the image to ensure that important details are not lost during the enhancement process. The joint loss function includes exposure consistency loss, color consistency loss, and spatial consistency loss, which respectively ensure the consistency of the image's exposure, color, and spatial structure, thereby avoiding the image being too dark, color distortion, and detail loss. By optimizing the joint loss function, the model can generate more realistic and clear images under different environmental conditions, improve the diversity and quality of the training dataset, and ultimately enhance the model's adaptability to low light and complex environments.

[0032] Furthermore, the local variance adaptive Gaussian filtering is used to enhance the image details, and the pixel gradients are classified and edge detection is performed by support vector machine including: Convert the enhanced image to a grayscale image, divide the grayscale image into fixed-size blocks, and use 16 A fixed square window of 16 pixels is used to scan and segment the grayscale image region by region to obtain multiple local sub-regions. The local grayscale variance value in each local sub-region is calculated and the grayscale variance values ​​of all local sub-regions are used as a one-dimensional distribution to measure the complexity of regional details. The current candidate grayscale variance threshold is set, and the optimal grayscale variance threshold is determined by calculating the grayscale histogram of the grayscale image and the Otsu algorithm formula, so that edges can be effectively detected under different lighting conditions. The formula is: , in, represents the optimal grayscale variance threshold, Respectively represent the number of pixels whose grayscale variance is less than and greater than or equal to the current candidate grayscale variance threshold, and Respectively represent the average gray value of the area where the pixel gray variance is less than and greater than or equal to the current candidate gray variance threshold, It represents the optimal grayscale variance threshold to be sought, which is used to distinguish the richness of regional details; The unique local Gaussian smoothing window size of each sub-region is determined according to the optimal grayscale variance threshold. The formula is: , in, Represents the grayscale image Row, No. The size of the Gaussian filter window corresponding to the column area, if , indicating that this area is rich in details, and the Gaussian filter window to be selected for this area is 3 3 pixels to preserve image details as much as possible. , indicating that the area is smooth, and the Gaussian filter window to be selected for this area is 7 7 pixels for better noise reduction; Based on the unique local Gaussian smoothing window size of each sub-region, a deterministic Gaussian filtering operation is performed on the grayscale value of each pixel position in each corresponding region of the original image, and the filtered grayscale value is used to replace the original grayscale value; After filtering each sub-region, the sub-regions are deterministically stitched back into a complete image of the original size according to the original image position; By calculating the gradient value of each pixel in the image (i.e., how fast the image brightness changes), the gradient amplitude and direction are calculated using the Canny edge detection algorithm, and edge detection is performed on each pixel to determine whether each pixel is an edge or a non-edge. Select the support vector machine (SVM) algorithm, use the RBF kernel (radial basis kernel) as the kernel function, extract the features of each pixel (e.g., gradient magnitude and direction) from the enhanced image as the input features of the training data set, mark the edge detection status of each pixel in the input features (edge ​​or non-edge), and these labels will be used as the target output of the SVM classifier. Use the labeled training data set to train the model by minimizing the loss function of the support vector machine. By using the gradient descent method, the optimal hyperplane and bias are solved to minimize the classification error, thereby obtaining the best classification model; The features of each pixel of the image are input into the trained support vector machine model for classification. The SVM model will determine whether each pixel belongs to the edge area. Based on the output of the support vector machine model, the edge probability of each pixel in the image is adjusted to make the detected edge more accurate. The optimized output of SVM will improve the accuracy of edge detection and reduce false detection and missed detection. Especially in areas with fuzzy edge transitions or more noise, SVM can help identify the real edges more accurately.

[0033] This method combines local variance adaptive Gaussian filtering with support vector machine (SVM) classification for image detail enhancement and edge detection, significantly improving the accuracy of image edge recognition. In the process of detail enhancement, local grayscale variance is used to measure the detail complexity of each sub-region, so as to assign the most appropriate Gaussian smoothing window to each region, which can both retain details and suppress noise. The SVM algorithm uses gradient amplitude and direction as features to perform edge classification and optimize the accuracy of edge detection. By minimizing the loss function of SVM, the model can accurately distinguish edge and non-edge pixels, reduce false detection and missed detection, especially in areas with blurred edge transitions or more noise, ensuring that edge detection is more accurate and reliable. This method can still stably and effectively identify image edges under complex lighting conditions and has strong robustness.

[0034] S4, ResNet50 is used to extract features, combined with RPN and non-maximum suppression to accurately locate candidate boxes, and object detection and recognition are performed through classification regression and visualized display; Specifically, ResNet50 is used to extract features, and RPN and non-maximum suppression are combined to accurately locate candidate boxes. Object detection and recognition are performed through classification regression and visualized display, including: Use the pre-trained ResNet50 model as the feature extractor. ResNet50 is a 50-layer convolutional neural network with strong image feature extraction capabilities. It can extract multi-level features from images through multi-layer convolution, pooling, and residual connections. Use global average pooling to pool the features of each channel to obtain a one-dimensional feature vector to describe the objects in the image. Use Faster R-CNN's Region Proposal Network (RPN) to generate object candidate boxes. RPN predicts the candidate box at each position by sliding a small window on the convolutional feature map, and scores each candidate box to indicate the probability of containing the object. Use the non-maximum suppression (NMS) algorithm to select the best candidate box from multiple object candidate boxes, use the Softmax classifier to classify each candidate box, and determine the category of each candidate box by setting a threshold. If the probability of a category exceeds the set threshold, the object is considered to belong to that category, otherwise it is considered that the object does not exist or belongs to the "background" category. Perform regression operations on each candidate box to predict the specific position and size of the object in the box, visualize the object detection and recognition results, and generate the final output.

[0035] By combining the pre-trained ResNet50 feature extractor, RPN and non-maximum suppression algorithm, this method can accurately extract multi-level features from the image, generate high-quality object candidate boxes and perform accurate classification and regression. ResNet50 provides powerful image feature extraction capabilities, and uses global average pooling to effectively reduce the number of parameters and improve model efficiency. RPN can generate potential object areas by sliding windows and assign confidence scores to each candidate box. Non-maximum suppression ensures the optimization of candidate boxes and avoids interference from duplicate boxes. Through the Softmax classifier and regression operation, the model can accurately identify object categories and make detailed predictions on the specific locations of objects. Combining these methods for object detection and recognition not only improves the detection accuracy, but also can clearly display the location and category of objects in the image, achieving efficient and accurate object detection and visualization.

[0036] S5. Store all data in the database and manage it; Specifically, storing and managing all data in the database includes: Select a relational database to manage data and relationship analysis results, design the database table structure to store different types of data, set up regular backup tasks, back up all data in the database, manage the permissions of database users, and encrypt and store static data.

[0037] Storing and managing all data in a database can bring about multiple beneficial effects. Relational databases can effectively manage different types of data and the relationships between them, ensuring data consistency, integrity, and queryability. By reasonably designing the database table structure, data storage and retrieval performance can be optimized, and data management efficiency can be improved. Regular backup tasks ensure data security and can quickly restore the system in the event of a failure to prevent data loss. Database user authority management ensures that only authorized users can access sensitive data, improving data security. Encrypted storage of static data can effectively prevent data leakage and ensure the confidentiality of data during storage. These measures can improve data security, reliability, and management efficiency, while enhancing the system's fault tolerance and maintainability.

[0038] This embodiment also provides a data processing system based on an image acquisition device, including: A data acquisition module, used for acquiring multimodal images from an image acquisition device and performing preprocessing; The model building module is used to integrate multimodal images into a training dataset, build a CNN-VAE to generate low-light images of multimodal images, and use a GAN model with an anti-attention module and a dual discriminator structure, combined with parallel expansion and dilated convolution to expand the training dataset; Image enhancement module, used to enhance images in the training dataset through a dual-branch U-Net; Edge detection module, used to enhance image details using local variance adaptive Gaussian filtering, classify pixel gradients and perform edge detection through support vector machine; Feature extraction module, which uses ResNet50 to extract features and combines RPN and non-maximum suppression to accurately locate candidate boxes; The target recognition module is used to detect and identify objects through classification and regression and to visualize them; The data storage module is used to store and manage all data in the database.

[0039] This embodiment also provides a computer device, which is suitable for the case of a data processing method based on an image acquisition device, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to implement the data processing method based on the image acquisition device proposed in the above embodiment. The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0040] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by the processor, the data processing method based on the image acquisition device proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0041] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data processing method based on an image acquisition device, characterized in that: include: Acquire multimodal images from an image acquisition device and perform preprocessing; The multimodal image includes an RGB image and an infrared image; Integrate multimodal images into training datasets, build CNN-VAE to generate low-light images of multimodal images, use the GAN model with anti-attention module and dual discriminator structure, and combine parallel expansion and dilated convolution to expand the training dataset; The images in the training dataset are enhanced by using a dual-branch U-Net, the image details are enhanced by using a local variance adaptive Gaussian filter, and the pixel gradients are classified and edge detection is performed by a support vector machine. ResNet50 is used to extract features, and RPN and non-maximum suppression are combined to accurately locate candidate boxes. Object detection and recognition are performed through classification regression and visualized. All data is stored and managed in the database.

2. The data processing method based on the image acquisition device according to claim 1, characterized in that: The method of constructing a CNN-VAE to generate low-light images of multimodal images, using a GAN model with an anti-attention module and a dual discriminator structure, and combining parallel expansion of the dilated convolution to expand the training data set includes: Use random rotation to simulate different directions of the preprocessed image, adjust the brightness of the multimodal image through histogram equalization and integrate it into the training data set, map the training data set to the latent space through the variational autoencoder, and sample the latent space data from the latent space , by maximizing the variational lower bound, the variational autoencoder learns the distribution of the latent space and generates low-light images by minimizing the loss function, and the generated low-light images are used as new samples to expand the training dataset; Use the generative adversarial network to expand the training dataset, define the anti-attention module, build a dual discriminator structure to extract the global and local features of the image and add them to each layer in the generator, use global average pooling to encode the spatial information of the multimodal images in the training dataset, perform nonlinear transformation through the Sigmoid activation function, generate the anti-attention map and suppress unnecessary areas by performing the reverse operation to generate the image; A parallel extended convolution module is integrated into the generative adversarial network. By using convolutions with different dilation rates in parallel, the receptive field of the generative adversarial network is expanded and image enhancement is performed, and the final enhanced data set is output.

3. The data processing method based on the image acquisition device according to claim 2, characterized in that: The construction of the dual discriminator structure to extract the global features and local features of the image includes: The anti-attention module extracts the global and local features of the image, adjusts the weights of the feature channels according to the importance of the global and local features, uses PatchGAN as the local discriminator, and combines it with the global discriminator for training. The real image and the generated image are used to train the discriminator, calculate the discriminator loss, and use the Adam optimizer to train the model until convergence.

4. The data processing method based on the image acquisition device according to claim 1, characterized in that: The method of enhancing the images in the training data set by using the dual-branch U-Net includes: The U-Net architecture is used as the joint learning framework and the anti-attention module is integrated for each convolutional layer. The RGB images and infrared images in the final enhanced dataset are respectively input into two parallel U-Net branches to generate enhanced images. The local features of the image are retained through skip connections. The parameters of U-Net are updated using the joint loss function. The optimizer is defined, the gradient is calculated and updated, and the U-Net model is optimized to obtain the final enhanced image.

5. The data processing method based on the image acquisition device according to claim 1, characterized in that: The method of enhancing image details by using local variance adaptive Gaussian filtering and classifying pixel gradients and performing edge detection by using a support vector machine comprises: The enhanced image is converted into a grayscale image, and the grayscale image is divided into fixed-size blocks to obtain multiple local sub-regions. The local grayscale variance value in each local sub-region is calculated and the grayscale variance values ​​of all local sub-regions are used as a one-dimensional distribution. The optimal grayscale variance threshold is determined by calculating the grayscale histogram of the grayscale image and the Otsu algorithm formula. Determine the unique local Gaussian smoothing window size for each sub-region according to the optimal grayscale variance threshold, perform deterministic Gaussian filtering on the grayscale value of each pixel position in each corresponding region of the original image, replace the original grayscale value with the filtered grayscale value, and deterministically splice each sub-region back to the original size of the complete image according to the original image position after filtering. Calculate the gradient value of each pixel in the image, calculate the gradient amplitude and direction through the Canny edge detection algorithm, and perform edge detection on each pixel; Select the support vector machine algorithm, use the RBF kernel as the kernel function, extract the features of each pixel from the enhanced image as the input features of the training data set, mark the edge detection of each pixel in the input features, use the labeled training data set, train the model by minimizing the loss function of the support vector machine, and solve the optimal hyperplane and bias by using the gradient descent method; The features of each pixel of the image are input into the trained support vector machine model for classification, and the edge probability of each pixel in the image is adjusted based on the output of the support vector machine model.

6. The data processing method based on the image acquisition device according to claim 1, characterized in that: The method uses ResNet50 to extract features, combines RPN and non-maximum suppression to accurately locate candidate boxes, and performs object detection and recognition through classification regression and visual display, including: Use the pre-trained ResNet50 model as the feature extractor; Use global average pooling to pool the features of each channel to obtain a one-dimensional feature vector, use Faster R-CNN's Region Proposal Network to generate object candidate boxes, use the non-maximum suppression algorithm to select the best candidate box from multiple object candidate boxes, use the Softmax classifier to classify each candidate box, determine the category of each candidate box by setting a threshold, perform regression on each candidate box, predict the specific position and size of the object in the box, visualize the object detection and recognition results, and generate the final output.

7. The data processing method based on the image acquisition device according to claim 1, characterized in that: The storing and managing of all data in the database includes: Select a relational database to manage data and relationship analysis results, design the database table structure to store different types of data, set up regular backup tasks, back up all data in the database, manage the permissions of database users, and encrypt and store static data.

8. A data processing system based on an image acquisition device based on the data processing method based on an image acquisition device according to any one of claims 1 to 7, characterized in that: include: A data acquisition module, used for acquiring multimodal images from an image acquisition device and performing preprocessing; The model building module is used to integrate multimodal images into a training dataset, build a CNN-VAE to generate low-light images of multimodal images, and use a GAN model with an anti-attention module and a dual discriminator structure, combined with parallel expansion and dilated convolution to expand the training dataset; Image enhancement module, used to enhance images in the training dataset through a dual-branch U-Net; Edge detection module, used to enhance image details using local variance adaptive Gaussian filtering, classify pixel gradients and perform edge detection through support vector machine; Feature extraction module, which uses ResNet50 to extract features and combines RPN and non-maximum suppression to accurately locate candidate boxes; The target recognition module is used to detect and identify objects through classification and regression and to visualize them; The data storage module is used to store and manage all data in the database.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the data processing method based on the image acquisition device described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data processing method based on an image acquisition device described in any one of claims 1 to 7 are implemented.

Images