Picture quality inspection model fine tuning method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and finance, and relates to a picture quality inspection model fine tuning method, which comprises the following steps: preprocessing a picture data set, labeling picture tags, obtaining an enhanced picture set, training a pre-constructed initial picture quality inspection model based on the enhanced picture set, and performing fine tuning on the picture quality inspection model in the training process. And adjusting network parameters of an input space expansion layer and an output layer of the initial picture quality inspection model until a final picture quality inspection model is obtained, and inputting a to-be-detected picture into the picture quality inspection model to obtain a picture category. The invention further provides a picture quality inspection model fine adjustment device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the picture data set can be stored in a block chain. The method can effectively shorten the training time, improve the development efficiency, and improve the prediction precision and reliability of the model.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a method, device, computer equipment and storage medium for fine-tuning an image quality inspection model. Background Art

[0002] In the field of financial services, with the acceleration of the digitalization process, the financial intelligent customer service system has become a key link in improving service quality and enhancing user experience. Among them, the image quality inspection model, as an important part of the intelligent customer service system, plays a vital role in effectively identifying and filtering false, fraudulent or misleading financial-related images. The application of this model not only ensures the credibility of customer service, allowing customers to obtain accurate and authentic financial information, thereby improving the overall user experience, but also significantly reduces the financial risks caused by false information, ensures the security and privacy protection of customer data, and effectively maintains the healthy and orderly development of the financial industry.

[0003] Furthermore, the image quality inspection model of financial intelligent customer service is an important technical means for financial institutions to comply with relevant laws and regulations and regulatory requirements. By automatically identifying and filtering non-compliant images, financial institutions can proactively avoid potential legal risks, improve the legal and compliant operation level of enterprises, and thus shape a good corporate image and enhance market credibility.

[0004] However, as financial service scenarios become increasingly rich and complex, the types and number of images circulating in customer service systems continue to rise. At the same time, new types of illegal images are emerging one after another, which puts higher demands on the accuracy and adaptability of image quality inspection models. Traditional image quality inspection methods often rely on fixed algorithms and rule libraries, which are difficult to flexibly respond to changing image types and violation patterns, and therefore require frequent manual adjustments and optimizations. This process is not only time-consuming and labor-intensive, but also easily consumes a large amount of computing resources, resulting in increased operation and maintenance costs. It may also reduce the quality inspection effect due to untimely adjustments, affecting the overall performance of the intelligent customer service system. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for fine-tuning an image quality inspection model, so as to solve the technical problems in the related art that the image quality inspection algorithm is difficult to flexibly respond to the growing and changing types of images, consumes a large amount of computing resources, and easily causes untimely adjustments and reduces the quality inspection effect.

[0006] In order to solve the above technical problems, the present application embodiment provides a method for fine-tuning an image quality inspection model, which adopts the following technical solution:

[0007] Obtaining a picture data set, preprocessing the picture data set and labeling the pictures to obtain an enhanced picture set, and dividing the enhanced picture set into a training set and a test set;

[0008] Inputting the training set into a pre-built initial image quality inspection model, wherein the initial image quality inspection model includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer;

[0009] Segmenting each enhanced image in the training set through the image segmentation layer to obtain an image segmentation set;

[0010] Performing vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector;

[0011] The block set embedding vector and the introduced hint vector are concatenated through the input space expansion layer to obtain a picture expansion feature vector;

[0012] Inputting the image extension feature vector into the main network, fusing the image extension feature vector through an attention mechanism to obtain an image fusion feature vector;

[0013] Inputting the image fusion feature vector into the output layer for calculation and output, to obtain the image prediction category;

[0014] Based on a preset loss function, a loss value between the image prediction category and the image label is calculated, and network parameters of the input space expansion layer and the output layer are adjusted according to the loss value, and iterative training is continued to obtain a model to be tested;

[0015] Using the test set to test the model to be tested to obtain a test result, and when the test result meets the expected conditions, using the model to be tested as the final image quality inspection model;

[0016] Obtain a picture to be detected, input the picture to be detected into the picture quality inspection model, and obtain the picture category.

[0017] In order to solve the above technical problems, the embodiment of the present application further provides a device for fine-tuning an image quality inspection model, which adopts the following technical solution:

[0018] An acquisition module is used to acquire a picture data set, preprocess the picture data set and label the pictures to obtain an enhanced picture set, and divide the enhanced picture set into a training set and a test set;

[0019] An input module, used to input the training set into a pre-built initial image quality inspection model, wherein the initial image quality inspection model includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer;

[0020] A segmentation module, used to segment each enhanced image in the training set according to the image segmentation layer to obtain a set of image blocks;

[0021] An embedding module, used for performing vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector;

[0022] An expansion module, used to concatenate the block set embedding vector and the introduced prompt vector through the input space expansion layer to obtain a picture expansion feature vector;

[0023] A feature extraction module, used for inputting the image extension feature vector into the main network, fusing the image extension feature vector through an attention mechanism, and obtaining an image fusion feature vector;

[0024] An output module, used for inputting the image fusion feature vector into the output layer for calculation and output, to obtain a predicted image category;

[0025] An iterative module, used to calculate the loss value between the image prediction category and the image label based on a preset loss function, and adjust the network parameters of the input space expansion layer and the output layer according to the loss value, and continue iterative training to obtain a model to be tested;

[0026] A testing module, used to test the model to be tested using the test set to obtain a test result, and when the test result meets the expected conditions, use the model to be tested as the final image quality inspection model;

[0027] The classification module is used to obtain the image to be detected, input the image to be detected into the image quality inspection model, and obtain the image category.

[0028] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0029] The computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned image quality inspection model fine-tuning method when executing the computer-readable instructions.

[0030] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0031] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the image quality inspection model fine-tuning method as described above are implemented.

[0032] Compared with the prior art, this application has the following beneficial effects:

[0033] The present application provides a method for fine-tuning an image quality inspection model, which obtains an enhanced image set by preprocessing an image data set and annotating image labels, thereby improving image quality and improving the reliability of subsequent feature extraction; a pre-constructed initial image quality inspection model is trained based on the enhanced image set, and during the training process, only the network parameters of the input space expansion layer and the output layer are adjusted, so that the training process of the model is more efficient and faster, which can effectively shorten the training time and improve development efficiency; at the same time, a pre-trained main network is adopted, and the main network parameters are kept unchanged during the training process, which can significantly reduce the number of parameters required for the model, reduce the computational burden and memory usage of the model, thereby improving the running speed and efficiency of the model, and in addition, it can also improve the prediction accuracy and reliability of the model, so as to flexibly respond to the growing and changing image types and violation types, quickly identify and filter illegal images, thereby improving the effect and value of the model in practical applications, and providing users with a better service experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0036] Figure 2 is a flowchart of an embodiment of a method for fine-tuning an image quality inspection model according to the present application;

[0037] Figure 3 is a structural schematic diagram of an embodiment of a device for fine-tuning an image quality inspection model according to the present application;

[0038] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0040] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0043] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0044] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0045] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0046] It should be noted that the image quality inspection model fine-tuning method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the image quality inspection model fine-tuning device is generally set in the server / terminal device.

[0047] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0048] Continue to refer Figure 2 , shows a flowchart of an embodiment of a method for fine-tuning an image quality inspection model according to the present application, comprising the following steps:

[0049] Step S201, obtaining a picture data set, preprocessing the picture data set and labeling the picture labels to obtain an enhanced picture set, and dividing the enhanced picture set into a training set and a test set.

[0050] The image dataset can be obtained from the database of the financial service customer service system, or from the third-party social platform of the insurance business to collect image data from customer communication. The obtained image dataset is preprocessed, including denoising, image enhancement, filtering and other processing methods.

[0051] In this embodiment, the image quality inspection model fine-tuning method is executed on the electronic device (eg Figure 1 The server / terminal device shown in the figure can obtain the image data set through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0052] The original images in the image dataset are cleaned to remove abnormal data and noise data. Image augmentation is a data enhancement technology. By performing image transformations such as rotation, flipping, translation, scaling, and noise injection on the images, more training samples can be generated to achieve image augmentation, which helps to improve the generalization ability of the model. Among them, rotation is to randomly rotate the image to a certain angle; flipping is to flip the image horizontally or vertically; translation is to apply a random translation transformation on the image; scaling is to change the size of the image; and noise injection is to add random noise to the image. The frequency domain homomorphic filtering method is used to screen the image degradation after image enhancement. Among them, homomorphic filtering is a filtering method used to adjust the brightness and contrast of the image, which is particularly suitable for images with uneven lighting. Frequency domain homomorphic filtering decomposes the image in the frequency domain, then filters each frequency component, and finally re-synthesizes the filtered results into a picture.

[0053] The preprocessed image dataset is annotated, and each image is annotated with an image label. The image label contains the image type and violation type. The image types include insurance product promotional posters, bank wealth management product promotional posters, stock market charts, fund fixed investment yield curves, financial product images, financial service usage scenario images (such as online payment, transfer, loan application and other scenes), etc.; violation types include illegal advertising law provisions, infringement of personal information rights, unauthorized financial services, misleading publicity, violation of cross-border financial service regulations, etc.

[0054] The annotated enhanced image set is randomly divided into a training set and a test set according to a preset ratio. The preset ratio can be set according to actual needs. For example, the preset ratio is 8:2, that is, the ratio of the training set to the test set is 8:2. If the training set has a total of N images, the training set is represented as Among them, X i represents the i-th picture, Y i Indicates the corresponding image label.

[0055] It should be emphasized that in order to further ensure the privacy and security of the image dataset, the above-mentioned image dataset can also be stored in a blockchain node.

[0056] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0057] Step S202: input the training set into a pre-built initial image quality inspection model, wherein the initial image quality inspection model includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer.

[0058] In this embodiment, a training set is used to train an initial image quality inspection model, which includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer. The images in the training set are processed in sequence by the image segmentation layer, the image embedding layer, the input space expansion layer, the pre-trained main network and the output layer to output the category prediction value of the model.

[0059] Among them, the image segmentation layer is used to divide the image into blocks; the image embedding layer is used to convert the image blocks into embedding vectors that the model can understand; the input space expansion layer introduces a learnable prompt vector pool to spatially expand the image embedding vector to enrich the feature representation; the main network is used to calculate the image expansion feature vector through the attention mechanism, so that the feature vector contains more or can better express the characteristics of the area of ​​interest of the model, thereby improving the detection performance of the model; the output layer is used to make the final prediction and classification of the input image fusion feature vector, and convert the internal representation of the model into a final result that is easy to understand and use for output.

[0060] Step S203, segment each enhanced image in the training set by the image segmentation layer to obtain a set of image segments.

[0061] In this embodiment, each picture is segmented by using a segmentation function through the picture segmentation layer. For example, the i-th picture is segmented into m picture blocks, which are expressed as follows:

[0062] {I j ∈R 3×h×w |j∈N,1≤j≤m}=Patch(X i )

[0063] Where Patch(·) represents the segmentation function; h and w represent the height and width of the image block respectively; I j Represents the j-th image block of the i-th image.

[0064] In some optional implementations, the step of segmenting each enhanced image in the training set by the image segmentation layer to obtain a set of image segments includes:

[0065] A sliding window is used to slide on each enhanced image with a preset window size and a preset step size to obtain multiple initial image blocks;

[0066] The multiple initial image blocks are segmented by an image segmentation algorithm to obtain a final set of image blocks.

[0067] In this embodiment, the enhanced image is firstly divided into blocks by a sliding window, wherein the preset window size and the preset step size can be set according to actual needs. For example, a window size of 64×64 and a step size of 32 pixels are slid on the enhanced image to obtain multiple initial image blocks; then, the multiple initial image blocks are further segmented by an image segmentation algorithm to obtain finer-grained image blocks.

[0068] The image segmentation algorithm can adopt the Mask R-CNN model, Unet network model, etc., among which the Mask R-CNN model includes a feature extraction layer, a region proposal layer, an alignment layer, a multi-task branch layer and an output layer; the feature extraction layer performs feature extraction on each initial image block to obtain the image semantic features; the image semantic features are input into the region proposal layer to extract the region of interest to obtain the candidate region; the candidate region is input into the alignment layer to obtain the aligned region of interest; the aligned region of interest is classified, bounding box regressed and segmented through the multi-task branch layer, and the category, bounding box and instance mask images are obtained accordingly, and the instance mask image is output as the final image block.

[0069] The Unet network model includes an encoder, a decoder and a connection layer. The encoder performs multi-level feature extraction on each initial image block to obtain a hierarchical image feature map and a down-sampled feature map, and transmits the hierarchical image feature map to the decoder; the connection layer performs convolution and up-sampling on the down-sampled feature map to obtain enhanced up-sampling features, and transmits the enhanced up-sampling features to the decoder; the decoder performs multi-level up-sampling operations on the enhanced up-sampling features and the hierarchical image feature map to output the final set of image blocks.

[0070] By dividing the enhanced image into a set of image blocks, the granularity of feature extraction can be improved, which helps to more accurately extract local features of the image, reduce noise interference, and improve recognition accuracy. At the same time, it can reduce computational complexity, improve image processing efficiency, and enhance the generalization ability of the model.

[0071] Step S204: perform vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector.

[0072] In this embodiment, each image block in the image block set is converted into an image embedding vector through the image embedding layer. For example, the pre-trained Img2Vec model is used to convert m image blocks into image embedding vectors respectively, which are expressed as follows:

[0073]

[0074] in, represents the j-th embedding vector of layer 0 (i.e., the j-th image block); m vectors are combined into an input set vector (i.e., the block set embedding vector) expressed as

[0075]

[0076] In some optional implementations of this embodiment, the step of performing vector conversion on each picture block in the picture block set through the picture embedding layer to obtain the block set embedding vector includes:

[0077] Input the image block set into the image embedding layer for feature extraction to obtain the feature vector of each image block;

[0078] The feature vector is mapped into a low-dimensional space through principal component analysis to obtain the embedded vector representation of each image block;

[0079] According to the position information of each image block in the enhanced image, the embedded vector representation is concatenated in a preset order to obtain the block set embedding vector.

[0080] In this embodiment, a pre-trained convolutional neural network such as ResNet-50 can be used to extract the visual feature vector of each image block in the image block set through convolution and pooling operations, including color histogram, local binary pattern (LBP) texture feature and Hu invariant rectangular feature. Then, principal component analysis (PCA) is used to map the visual feature vector to a low-dimensional space to obtain an embedded vector representation of each image block. Finally, according to the position information of the image block in the enhanced image, the embedded vector representation of each enhanced image is spliced ​​into a vector sequence in a left-to-right and top-to-bottom order, that is, the embedded vector of the block set of each enhanced image is obtained.

[0081] By converting the image blocks into embedding vectors, it is possible to capture the local features in each block more carefully and process multiple image blocks in parallel, thereby improving the accuracy and efficiency of feature extraction; the image blocks contain different parts and features of the image, which can increase the diversity of the model. The embedding operation can further extract these diverse features, enhance the generalization ability of the model, and thus improve the accuracy of image classification.

[0082] Step S205 , concatenating the block set embedding vector and the introduced hint vector through the input space expansion layer to obtain the image expansion feature vector.

[0083] In order to reduce the computational overhead and ensure that the pre-trained backbone network remains unchanged, a learnable hint vector pool P is introduced in the input space, where the hint vector pool is represented as P = {p k ∈R d |k∈N,1≤k≤S}, there are a total of S hint vectors in the hint vector pool.

[0084] Specifically, the block set embedding vector is input into the input space expansion layer, and a preset prompt vector pool is obtained through the input space expansion layer, where the prompt vector pool contains S vectors, where S is a positive integer; k pre-trained prompt vectors are obtained from the prompt vector pool, where k is a positive integer and is less than or equal to S; the block set embedding vector is concatenated with the k prompt vectors to obtain a picture extension feature vector. The picture extension feature vector is represented as [x1, P, E0].

[0085] By introducing the hint vector, we actually introduce additional information or constraints on the basis of the original feature space, thereby expanding the feature space and enabling the model to capture more features related to image classification, thereby improving classification accuracy; at the same time, we enhance the generalization ability of the model, improve computational efficiency, and optimize the model training process.

[0086] Step S206: input the image extension feature vector into the main network, fuse the image extension feature vector through the attention mechanism, and obtain the image fusion feature vector.

[0087] In this embodiment, the backbone network fuses the extended feature vector of the image through the attention mechanism, which helps to extract local and global features of the image from different subspaces. Among them, the backbone network is usually composed of multiple levels or modules, which focus on gradually extracting abstract features in the data. For example, multiple convolutional layers are used to gradually extract abstract features in the extended feature vector of the image, including shape, texture, color, etc.; features at different levels are fused through operations to obtain fused features, and the fused features are transferred between layers through residual connections or jump connections to retain more feature information and avoid the gradient disappearance problem, and finally the features are output through the last layer.

[0088] In some optional implementations of this embodiment, the step of inputting the image extension feature vector into the main network, fusing the image extension feature vector through the attention mechanism, and obtaining the image fusion feature vector includes:

[0089] The image expanded feature vector is input into the main network, which includes N layers of attention fusion layers;

[0090] Through the first attention fusion layer, the multi-head attention mechanism is used to calculate the attention weight of the image extension feature vector to obtain the first layer fusion feature vector;

[0091] According to the first layer fusion feature vector, the fusion feature vector is transmitted and updated layer by layer through the calculation process of the main network until the Nth layer fusion feature vector is obtained;

[0092] The Nth layer fused feature vector is output as the image fused feature vector.

[0093] In this embodiment, the attention weight of the image extended feature vector calculated using the multi-head attention mechanism is expressed as follows:

[0094] [x1,Z1,E1]=L1([x0,P,E0]);

[0095] Among them, L1(·) represents the attention mechanism operator of the first attention fusion layer; x0 represents the image category information vector.

[0096] The calculation process of the middle layer of the main network is as follows:

[0097] [x i ,Z i ,E i ]=L i ([x i-1 ,Z i-1 ,E i-1 ]).

[0098] By extracting features from the image extension feature vector through the backbone network, we can make full use of the powerful representation ability of the backbone network, accurately capture the key information in the image extension feature vector, and use the attention mechanism to weight the image extension feature vector, so that the model pays more attention to the key areas in the image, making the model more adaptable to complex image types, thereby enhancing the generalization ability of the model and further improving the accuracy and robustness of the model.

[0099] Step S207, input the picture fusion feature vector into the output layer for calculation and output, and obtain the picture prediction category.

[0100] In this embodiment, the image fusion feature vector is input into the output layer, the output layer is a classifier, and multi-classification is performed through the Softmax function to predict the probability that the current image belongs to each category, obtain the predicted value, and determine the predicted category of the image according to the predicted value.

[0101] The final output is as follows:

[0102] y i =Head(x N );

[0103] In the formula, the backbone network has a total of N layers, then x N represents the Nth layer fusion feature vector, y i represents the predicted value, and Head(·) represents a classifier.

[0104] Step S208, based on the preset loss function, calculate the loss value between the image prediction category and the image label, and adjust the network parameters of the input space expansion layer and the output layer according to the loss value, continue iterative training, and obtain the model to be tested.

[0105] Specifically, according to the image prediction category and image label, the cross entropy loss function is used to calculate the loss value; according to the loss value, the gradient of the preset loss function to the network parameters of the input space expansion layer and the output layer is calculated through the back propagation algorithm; according to the calculated gradient, the network parameters of the input space expansion layer and the output layer are updated to obtain new network parameter values; it is determined whether the current iteration meets the convergence conditions. If the convergence conditions are met, the final model to be tested is obtained; if the convergence conditions are not met, iterative training is continued until the model converges to obtain the final model to be tested.

[0106] Among them, the convergence condition is to reach the preset maximum number of iterations, or the current loss value no longer decreases compared to the loss value of the previous round.

[0107] When the convergence conditions are not met, use the new network parameter values ​​to predict the training set through the forward propagation algorithm to obtain the prediction value of the new round of iteration; use the new prediction value and the image label to calculate the cross entropy loss function to obtain the loss value of this round of iteration; compare the loss value of this round with the loss value of the previous round. If the loss function value decreases, it is considered that the model performance has been improved and the iterative optimization continues; if the loss value does not decrease for multiple consecutive rounds, the early stopping mechanism is triggered to stop the iteration, and the network parameters of the input space expansion layer and the output layer when the iteration is stopped are obtained as the final training result, and the model to be tested is output.

[0108] In this embodiment, the cross entropy loss function is expressed as follows:

[0109]

[0110] Among them, p i represents the predicted value, Y i represents the image label, and CE(·,·) represents the cross entropy function.

[0111] During the model training phase, the parameters of the main network are fixed, and focus is placed on training the prompt vector pool of the input space expansion layer and the network parameters of the output layer to achieve fine-tuning during the model training process, making the model training process more efficient and faster, effectively shortening the training time and improving development efficiency. At the same time, it can significantly reduce the number of parameters required for the model, which will reduce the computational burden and memory usage of the model, thereby improving the running speed and efficiency of the model.

[0112] Step S209: Use the test set to test the model to be tested to obtain a test result. When the test result meets the expected conditions, the model to be tested is used as the final image quality inspection model.

[0113] When testing the model, the test set is input into the model to be tested to obtain the model prediction category; the evaluation index is calculated based on the model prediction category and the corresponding image label to obtain the test result. Among them, the evaluation index includes accuracy, precision, recall rate, F1 value, etc. The actual performance of the model can be comprehensively judged by the evaluation index, which can improve the prediction accuracy and reliability of the model, thereby improving the effect and value of the model in practical applications and providing users with a better service experience.

[0114] Step S210, obtaining a picture to be detected, inputting the picture to be detected into a picture quality inspection model, and obtaining a picture category.

[0115] The trained image quality inspection model is used to quickly identify and classify the images to be inspected, determine the violation category to which the images to be inspected belong, and complete the image quality inspection. The entire process uses the image quality inspection model to achieve end-to-end image detection and classification, which can effectively improve the accuracy and efficiency of image quality inspection.

[0116] In some optional implementations, the image quality inspection model is regularly adjusted and optimized to adapt to complex and changing financial service scenarios.

[0117] This application obtains an enhanced picture set by preprocessing the picture data set and annotating picture labels, which can improve the picture quality and the reliability of subsequent feature extraction; based on the pre-constructed initial picture quality inspection model trained on the enhanced picture set, only the network parameters of the input space expansion layer and the output layer are adjusted during the training process, so that the model training process is more efficient and faster, which can effectively shorten the training time and improve the development efficiency; at the same time, the pre-trained main network is adopted, and the main network parameters are kept unchanged during the training process, which can significantly reduce the number of parameters required for the model, reduce the computational burden and memory usage of the model, thereby improving the running speed and efficiency of the model. In addition, it can also improve the prediction accuracy and reliability of the model, so as to flexibly respond to the growing and changing picture types and violation types, quickly identify and filter illegal pictures, thereby improving the effect and value of the model in practical applications, and providing users with a better service experience.

[0118] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0119] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0121] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0122] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for fine-tuning an image quality inspection model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0123] like Figure 3As shown, the image quality inspection model fine-tuning device 300 described in this embodiment includes: an acquisition module 301, an input module 302, a segmentation module 303, an embedding module 304, an expansion module 305, a feature extraction module 306, an output module 307, an iteration module 308, a testing module 309 and a classification module 310. Among them:

[0124] The acquisition module 301 is used to acquire a picture data set, preprocess the picture data set and label the pictures to obtain an enhanced picture set, and divide the enhanced picture set into a training set and a test set;

[0125] The input module 302 is used to input the training set into a pre-built initial image quality inspection model, wherein the initial image quality inspection model includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer;

[0126] The segmentation module 303 is used to segment each enhanced image in the training set according to the image segmentation layer to obtain a set of image blocks;

[0127] The embedding module 304 is used to perform vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector;

[0128] The expansion module 305 is used to concatenate the block set embedding vector and the introduced prompt vector through the input space expansion layer to obtain a picture expansion feature vector;

[0129] The feature extraction module 306 is used to input the image extension feature vector into the main network, and fuse the image extension feature vector through the attention mechanism to obtain the image fusion feature vector;

[0130] The output module 307 is used to input the picture fusion feature vector into the output layer for calculation and output to obtain the picture prediction category;

[0131] The iteration module 308 is used to calculate the loss value between the image prediction category and the image label based on a preset loss function, and adjust the network parameters of the input space expansion layer and the output layer according to the loss value, and continue iterative training to obtain a model to be tested;

[0132] The testing module 309 is used to test the model to be tested using the test set to obtain a test result. When the test result meets the expected conditions, the model to be tested is used as the final image quality inspection model;

[0133] The classification module 310 is used to obtain the image to be detected, input the image to be detected into the image quality inspection model, and obtain the image category.

[0134] It should be emphasized that in order to further ensure the privacy and security of the image dataset, the above-mentioned image dataset can also be stored in a blockchain node.

[0135] Based on the above-mentioned image quality inspection model fine-tuning device 300, an enhanced image set is obtained by preprocessing the image data set and annotating image labels, which can improve the image quality and improve the reliability of subsequent feature extraction; based on the pre-constructed initial image quality inspection model of the enhanced image set training, during the training process, only the network parameters of the input space expansion layer and the output layer are adjusted, so that the training process of the model is more efficient and faster, which can effectively shorten the training time and improve the development efficiency; at the same time, using a pre-trained main network and keeping the main network parameters unchanged during the training process can significantly reduce the number of parameters required for the model, reduce the computational burden and memory usage of the model, thereby improving the running speed and efficiency of the model. In addition, it can also improve the prediction accuracy and reliability of the model, so as to flexibly respond to the growing and changing image types and violation types, quickly identify and filter illegal images, thereby improving the effect and value of the model in practical applications, and providing users with a better service experience.

[0136] In some optional implementations of this embodiment, the segmentation module 303 is further used to:

[0137] A sliding window is used to slide on each enhanced image with a preset window size and a preset step size to obtain multiple initial image blocks;

[0138] The plurality of initial picture blocks are segmented by an image segmentation algorithm to obtain a final picture block set.

[0139] By dividing the enhanced image into a set of image blocks, the granularity of feature extraction can be improved, which helps to more accurately extract local features of the image, reduce noise interference, and improve recognition accuracy. At the same time, it can reduce computational complexity, improve image processing efficiency, and enhance the generalization ability of the model.

[0140] In some optional implementations, the embedding module 304 is further configured to:

[0141] Inputting the picture block set into the picture embedding layer for feature extraction to obtain a feature vector for each picture block;

[0142] Performing low-dimensional space mapping on the feature vectors through principal component analysis to obtain an embedded vector representation of each of the image blocks;

[0143] According to the position information of each of the picture blocks in the enhanced picture, the embedding vector representations are concatenated in a preset order to obtain a block set embedding vector.

[0144] By converting the image blocks into embedded vectors, it is possible to capture the local features in each block more carefully and process multiple image blocks in parallel, thereby improving the accuracy and efficiency of feature extraction.

[0145] In some optional implementations, the expansion module 305 is further configured to:

[0146] Inputting the block set embedding vector into the input space expansion layer, obtaining a preset prompt vector pool through the input space expansion layer, wherein the prompt vector pool includes S vectors, where S is a positive integer;

[0147] Obtaining k pre-trained prompt vectors from the prompt vector pool, where k is a positive integer and is less than or equal to S;

[0148] The block set embedding vector is concatenated with the k prompt vectors to obtain a picture extension feature vector.

[0149] By introducing the hint vector, we actually introduce additional information or constraints on the basis of the original feature space, thereby expanding the feature space and enabling the model to capture more features related to image classification, thereby improving classification accuracy; at the same time, we enhance the generalization ability of the model, improve computational efficiency, and optimize the model training process.

[0150] In some optional implementations, the feature extraction module 306 is further used to:

[0151] Inputting the image expansion feature vector into the main network, wherein the main network includes N attention fusion layers;

[0152] Through the first attention fusion layer, a multi-head attention mechanism is used to calculate the attention weight of the image extended feature vector to obtain the first layer fusion feature vector;

[0153] According to the first layer of fused feature vectors, the fused feature vectors are transmitted and updated layer by layer through the calculation process of the main network until the Nth layer of fused feature vectors are obtained;

[0154] The N-th layer fused feature vector is output as the picture fused feature vector.

[0155] By extracting features from the image extension feature vector through the backbone network, we can make full use of the powerful representation ability of the backbone network, accurately capture the key information in the image extension feature vector, and use the attention mechanism to weight the image extension feature vector, so that the model pays more attention to the key areas in the image, making the model more adaptable to complex image types, thereby enhancing the generalization ability of the model and further improving the accuracy and robustness of the model.

[0156] In this embodiment, the iteration module 308 is further configured to:

[0157] Calculate the loss value using a cross entropy loss function according to the predicted category of the image and the image label;

[0158] According to the loss value, calculating the gradient of the preset loss function to the network parameters of the input space expansion layer and the output layer through a back propagation algorithm;

[0159] According to the calculated gradient, updating the network parameters of the input space expansion layer and the output layer to obtain new network parameter values;

[0160] Determine whether the current iteration meets the convergence condition. If the convergence condition is met, obtain the final model to be tested;

[0161] If the convergence condition is not met, continue iterative training until the model converges to obtain the final model to be tested.

[0162] By training the prompt vector pool of the input space expansion layer and the network parameters of the output layer, fine-tuning can be achieved during the model training process, making the model training process more efficient and faster, which can effectively shorten the training time and improve development efficiency.

[0163] In this embodiment, the testing module 309 is further used to:

[0164] Inputting the test set into the model to be tested to obtain the model prediction category;

[0165] The evaluation index is calculated according to the model prediction category and the corresponding image label to obtain the test result.

[0166] By comprehensively judging the actual performance of the model through evaluation indicators, the prediction accuracy and reliability of the model can be improved, thereby improving the effect and value of the model in practical applications and providing users with a better service experience.

[0167] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0168] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0169] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0170] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the image quality inspection model fine-tuning method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0171] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the image quality inspection model fine-tuning method.

[0172] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0173] By preprocessing the image data set and labeling the images, an enhanced image set is obtained, which can improve the image quality and the reliability of subsequent feature extraction. The pre-built initial image quality inspection model is trained based on the enhanced image set. During the training process, only the network parameters of the input space expansion layer and the output layer are adjusted, making the model training process more efficient and faster, which can effectively shorten the training time and improve development efficiency. At the same time, the pre-trained main network is used to keep the main network parameters unchanged during the training process, which can significantly reduce the number of parameters required for the model, reduce the computational burden and memory usage of the model, thereby improving the running speed and efficiency of the model. In addition, it can also improve the prediction accuracy and reliability of the model, so as to flexibly respond to the growing and changing image types and violation types, quickly identify and filter illegal images, thereby improving the effect and value of the model in practical applications, and providing users with a better service experience.

[0174] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned image quality inspection model fine-tuning method.

[0175] By preprocessing the image data set and labeling the images, an enhanced image set is obtained, which can improve the image quality and the reliability of subsequent feature extraction. The pre-built initial image quality inspection model is trained based on the enhanced image set. During the training process, only the network parameters of the input space expansion layer and the output layer are adjusted, making the model training process more efficient and faster, which can effectively shorten the training time and improve development efficiency. At the same time, the pre-trained main network is used to keep the main network parameters unchanged during the training process, which can significantly reduce the number of parameters required for the model, reduce the computational burden and memory usage of the model, thereby improving the running speed and efficiency of the model. In addition, it can also improve the prediction accuracy and reliability of the model, so as to flexibly respond to the growing and changing image types and violation types, quickly identify and filter illegal images, thereby improving the effect and value of the model in practical applications, and providing users with a better service experience.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for fine-tuning an image quality inspection model, characterized in that: The steps include: Obtaining a picture data set, preprocessing the picture data set and labeling the pictures to obtain an enhanced picture set, and dividing the enhanced picture set into a training set and a test set; Inputting the training set into a pre-built initial image quality inspection model, wherein the initial image quality inspection model includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer; Segmenting each enhanced image in the training set through the image segmentation layer to obtain an image segmentation set; Performing vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector; The block set embedding vector and the introduced hint vector are concatenated through the input space expansion layer to obtain a picture expansion feature vector; Inputting the image extension feature vector into the main network, fusing the image extension feature vector through an attention mechanism to obtain an image fusion feature vector; Inputting the image fusion feature vector into the output layer for calculation and output, to obtain the image prediction category; Based on a preset loss function, a loss value between the image prediction category and the image label is calculated, and network parameters of the input space expansion layer and the output layer are adjusted according to the loss value, and iterative training is continued to obtain a model to be tested; Using the test set to test the model to be tested to obtain a test result, and when the test result meets the expected conditions, using the model to be tested as the final image quality inspection model; Obtain a picture to be detected, input the picture to be detected into the picture quality inspection model, and obtain the picture category.

2. The image quality inspection model fine-tuning method according to claim 1, characterized in that: The step of segmenting each enhanced image in the training set by the image segmentation layer to obtain an image segmentation set includes: A sliding window is used to slide on each enhanced image with a preset window size and a preset step size to obtain multiple initial image blocks; The plurality of initial picture blocks are segmented by an image segmentation algorithm to obtain a final picture block set.

3. The image quality inspection model fine-tuning method according to claim 1, characterized in that: The step of performing vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector comprises: Inputting the picture block set into the picture embedding layer for feature extraction to obtain a feature vector for each picture block; Performing low-dimensional space mapping on the feature vectors through principal component analysis to obtain an embedded vector representation of each of the image blocks; According to the position information of each of the picture blocks in the enhanced picture, the embedding vector representations are concatenated in a preset order to obtain a block set embedding vector.

4. The image quality inspection model fine-tuning method according to claim 1, characterized in that: The step of concatenating the block set embedding vector and the introduced hint vector through the input space expansion layer to obtain the picture extension feature vector comprises: Inputting the block set embedding vector into the input space expansion layer, obtaining a preset prompt vector pool through the input space expansion layer, wherein the prompt vector pool includes S vectors, where S is a positive integer; Obtaining k pre-trained prompt vectors from the prompt vector pool, where k is a positive integer and is less than or equal to S; The block set embedding vector is concatenated with the k prompt vectors to obtain a picture extension feature vector.

5. The image quality inspection model fine-tuning method according to claim 1, characterized in that: The step of inputting the image extension feature vector into the main network and fusing the image extension feature vector through an attention mechanism to obtain the image fusion feature vector comprises: Inputting the image expansion feature vector into the main network, wherein the main network includes N attention fusion layers; Through the first attention fusion layer, a multi-head attention mechanism is used to calculate the attention weight of the image extended feature vector to obtain the first layer fusion feature vector; According to the first layer of fused feature vectors, the fused feature vectors are transmitted and updated layer by layer through the calculation process of the main network until the Nth layer of fused feature vectors are obtained; The N-th layer fused feature vector is output as the picture fused feature vector.

6. The image quality inspection model fine-tuning method according to claim 1, characterized in that: The step of calculating the loss value between the image prediction category and the image label based on a preset loss function, adjusting the network parameters of the input space expansion layer and the output layer according to the loss value, and continuing iterative training to obtain the model to be tested includes: According to the image prediction category and the image label, a cross entropy loss function is used to calculate a loss value; According to the loss value, calculating the gradient of the preset loss function to the network parameters of the input space expansion layer and the output layer through a back propagation algorithm; According to the calculated gradient, updating the network parameters of the input space expansion layer and the output layer to obtain new network parameter values; Determine whether the current iteration meets the convergence condition. If the convergence condition is met, obtain the final model to be tested; If the convergence condition is not met, continue iterative training until the model converges to obtain the final model to be tested.

7. The image quality inspection model fine-tuning method according to claim 1, characterized in that: The step of using the test set to test the model to be tested to obtain a test result comprises: Inputting the test set into the model to be tested to obtain the model prediction category; The evaluation index is calculated according to the model prediction category and the corresponding image label to obtain the test result.

8. A device for fine-tuning an image quality inspection model, characterized in that: include: An acquisition module is used to acquire a picture data set, preprocess the picture data set and label the pictures to obtain an enhanced picture set, and divide the enhanced picture set into a training set and a test set; An input module, used to input the training set into a pre-built initial image quality inspection model, wherein the initial image quality inspection model includes an image segmentation layer, an image embedding layer, an input space expansion layer, a pre-trained main network and an output layer; A segmentation module, used to segment each enhanced image in the training set according to the image segmentation layer to obtain a set of image blocks; An embedding module, used for performing vector conversion on each picture block in the picture block set through the picture embedding layer to obtain a block set embedding vector; An expansion module, used to concatenate the block set embedding vector and the introduced prompt vector through the input space expansion layer to obtain a picture expansion feature vector; A feature extraction module, used for inputting the image extension feature vector into the main network, fusing the image extension feature vector through an attention mechanism, and obtaining an image fusion feature vector; An output module, used for inputting the image fusion feature vector into the output layer for calculation and output, to obtain a predicted image category; An iterative module, used to calculate the loss value between the image prediction category and the image label based on a preset loss function, and adjust the network parameters of the input space expansion layer and the output layer according to the loss value, and continue iterative training to obtain a model to be tested; A testing module, used to test the model to be tested using the test set to obtain a test result, and when the test result meets the expected conditions, use the model to be tested as the final image quality inspection model; The classification module is used to obtain the image to be detected, input the image to be detected into the image quality inspection model, and obtain the image category.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the image quality inspection model fine-tuning method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the image quality inspection model fine-tuning method according to any one of claims 1 to 7 are implemented.

Images