Image processing method, device and medium
By acquiring the global color characteristics and semantic characteristics of the video frame, calculating the enhancement parameters for image enhancement, solving the problems of low processing efficiency and color jump phenomena in the prior art, and achieving efficient and stable image quality improvement.
Patent Information
- Application Number
- CN202411045693.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-28
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art has low processing efficiency and prone to color skipping during the video/image color adjustment process, resulting in poor image enhancement effect, especially when image information is large.
By obtaining the global color characteristics and image semantic characteristics of the target video frame collection, enhancing parameters are calculated, image enhancement is performed, calculation is reduced, and color jump risk is reduced, and image quality is improved.
It improves video data processing efficiency and image enhancement quality, reduces color jump phenomenon, and improves the stability and effect of image processing.
Smart Images

Figure CN119027323A_ABST
Abstract
Description
[0001] This application is a divisional application of the Chinese patent application filed with the China Patent Office on April 28, 2021, with application number 202110468372.7 and application name “Image processing method, device and medium”, all contents of which are incorporated by reference in this application. Technical Field
[0002] The present application relates to the field of artificial intelligence technology, and in particular to an image processing method, device and medium. Background Art
[0003] With the rapid development and widespread application of multimedia technology and network technology, people use videos / images more and more frequently in their daily lives and production activities. In the process of using videos / images, users have higher requirements for the quality of the videos / images they use.
[0004] At present, deep reinforcement learning methods can be used to adjust the color of videos / images to obtain enhanced videos / images to enhance the quality of videos / images. However, in the process of adjusting the color of videos / images, repeated adjustments are usually required to achieve the preset effect, which leads to low processing efficiency of videos / images. When there are large differences in the image information contained in the videos to be enhanced, the final image may have serious color jumps, resulting in poor image enhancement effects. Summary of the invention
[0005] The embodiments of the present application provide an image processing method, device, and medium, which can improve the processing efficiency of video data and improve the enhanced quality of video data.
[0006] An embodiment of the present application provides an image processing method, including:
[0007] A target video frame set is obtained from the video data to be processed; the video frames included in the target video frame set have the same attribute information;
[0008] Obtain a video frame to be processed from the target video frame set, obtain a global color feature corresponding to the video frame to be processed, and obtain an image semantic feature corresponding to the video frame to be processed;
[0009] According to the global color features and image semantic features, the enhancement parameters of the video frame to be processed in M image information dimensions are obtained; M is a positive integer;
[0010] According to the M enhancement parameters, image enhancement is performed on the video frames included in the target video frame set to obtain target image data corresponding to each video frame in the target video frame set.
[0011] Wherein, obtaining a target video frame set from the video data to be processed includes:
[0012] Obtain the video data to be processed, divide the video data to be processed into L video frames, and obtain the color histograms corresponding to the L video frames; the L video frames include video frames T having adjacent relationships. i and video frame T i+1 , L is a positive integer;
[0013] According to the video frame T i and video frame T i+1 The corresponding color histograms are obtained to obtain the video frame T i and video frame T i+1 The similarity distance between them; i is a positive integer less than L;
[0014] L video frames are divided according to similarity distances to obtain N video frame sets; the similarity distance between adjacent video frames in each video frame set is less than a distance threshold, and N is a positive integer;
[0015] Select any one video frame set from the N video frame sets as the target video frame set.
[0016] Wherein, obtaining the color histograms corresponding to L video frames respectively includes:
[0017] Divide the color space into P image color intervals and obtain video frame T in L video frames i ; P is a positive integer;
[0018] According to the video frame T i The image color space where the pixels are located in is counted, and the number of pixels contained in each image color interval is counted;
[0019] Generate video frame T according to the number of pixels corresponding to each of the P image color intervals i The corresponding color histogram.
[0020] Among them, obtaining the global color features corresponding to the video frame to be processed and obtaining the image semantic features corresponding to the video frame to be processed include:
[0021] Resize the video frame to be processed to obtain a candidate video frame with a target size, and obtain a global color feature corresponding to the video frame to be processed according to a color histogram corresponding to the candidate video frame;
[0022] The candidate video frame is input into the feature extraction model, and a convolution operation is performed on the candidate video frame according to the convolution layer in the feature extraction model to obtain the image convolution feature corresponding to the candidate video frame;
[0023] According to the residual layer in the feature extraction model, residual operation is performed on the image convolution features to obtain the image semantic features corresponding to the video frame to be processed.
[0024] Among them, according to the global color features and image semantic features, the enhancement parameters of the video frame to be processed in M image information dimensions are obtained, including:
[0025] The global color features and image semantic features are spliced to obtain the target image features;
[0026] The target image features are input into the target generation model, and the target generation model outputs the enhancement parameters of the target image features in M image information dimensions.
[0027] Among them, the enhancement parameters on the M image information dimensions include brightness enhancement parameters, contrast enhancement parameters and saturation enhancement parameters;
[0028] The target image features are input into the target generation model, and the target generation model outputs the enhancement parameters of the target image features in M image information dimensions, including:
[0029] Inputting the target image features into the target generation model, performing weighted processing on the target image features according to the weight matrix corresponding to the first regression network in the target generation model, and obtaining the brightness enhancement parameters corresponding to the video frame to be processed;
[0030] According to the weight matrix corresponding to the second regression network in the target generation model, the target image features are weighted to obtain the contrast enhancement parameters corresponding to the video frame to be processed;
[0031] According to the weight matrix corresponding to the third regression network in the target generation model, the target image features are weighted to obtain the saturation enhancement parameters corresponding to the video frame to be processed.
[0032] The target video frame set includes video frames T j ; j is a positive integer less than or equal to the number of video frames included in the target video frame set;
[0033] According to the M enhancement parameters, the video frames included in the target video frame set are enhanced to obtain target image data corresponding to each video frame in the target video frame set, including:
[0034] The brightness enhancement parameter is combined with the video frame T j The product of is determined as the video frame T j a corresponding first candidate video frame;
[0035] Obtain a global average pixel value corresponding to the first candidate video frame, obtain a first difference between the pixel value included in the first candidate video frame and the global average pixel value, and determine the video frame T according to the global average pixel value and the product of the first difference and the contrast enhancement coefficient. j a corresponding second candidate video frame;
[0036] Obtain the channel average pixel value corresponding to the second candidate video frame, obtain the second difference between the pixel value contained in the second candidate video frame and the channel average pixel value, and determine the video frame T according to the channel average pixel value and the product of the second difference and the saturation enhancement coefficient. j The corresponding target image data.
[0037] The method further includes:
[0038] Obtain sample video frames and label information corresponding to the sample video frames;
[0039] Inputting the sample global color features and the sample image semantic features corresponding to the sample video frame into the initial generation model, and outputting the prediction enhancement parameters corresponding to the sample video frame through the initial generation model;
[0040] Determine a loss function corresponding to the initial generation model according to label information corresponding to the sample video frame and prediction enhancement parameters corresponding to the sample video frame;
[0041] According to the loss function, the initial generation model is iteratively trained, and when the initial generation model meets the convergence condition, the initial generation model that meets the convergence condition is determined as the target generation model.
[0042] An embodiment of the present application provides an image processing device, including:
[0043] A first acquisition module is used to acquire a target video frame set from the video data to be processed; the video frames included in the target video frame set have the same attribute information;
[0044] The second acquisition module is used to acquire a to-be-processed video frame from the target video frame set, acquire a global color feature corresponding to the to-be-processed video frame, and acquire an image semantic feature corresponding to the to-be-processed video frame;
[0045] The third acquisition module is used to acquire enhancement parameters of the to-be-processed video frame in M image information dimensions respectively according to the global color features and the image semantic features; M is a positive integer;
[0046] The image enhancement module is used to perform image enhancement on the video frames included in the target video frame set according to M enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set.
[0047] Wherein, the first acquisition module includes:
[0048] The first acquisition unit is used to acquire the video data to be processed, divide the video data to be processed into L video frames, and acquire the color histograms corresponding to the L video frames respectively; the L video frames include video frames T having an adjacent relationship i and video frame T i+1 , L is a positive integer;
[0049] The second acquisition unit is used to obtain the video frame T i and video frame T i+1 The corresponding color histograms are obtained to obtain the video frame T i and video frame T i+1 The similarity distance between them; i is a positive integer less than L;
[0050] A division unit is used to divide the L video frames according to the similarity distance to obtain N video frame sets; the similarity distance between adjacent video frames in each video frame set is less than a distance threshold, and N is a positive integer;
[0051] The selection unit is used to select any one video frame set from the N video frame sets as a target video frame set.
[0052] The first acquisition unit is specifically used for:
[0053] Divide the color space into P image color intervals and obtain video frame T in L video frames i ; P is a positive integer;
[0054] According to the video frame T i The image color space where the pixels are located in is counted, and the number of pixels contained in each image color interval is counted;
[0055] Generate video frame T according to the number of pixels corresponding to each of the P image color intervals i The corresponding color histogram.
[0056] Wherein, the second acquisition module includes:
[0057] A third acquisition unit is used to resize the video frame to be processed to obtain a candidate video frame with a target size, and obtain a global color feature corresponding to the video frame to be processed according to a color histogram corresponding to the candidate video frame;
[0058] A convolution operation unit, used to input the candidate video frame into the feature extraction model, and perform a convolution operation on the candidate video frame according to the convolution layer in the feature extraction model to obtain the image convolution feature corresponding to the candidate video frame;
[0059] The residual operation unit is used to perform residual operation on the image convolution features according to the residual layer in the feature extraction model to obtain the image semantic features corresponding to the video frame to be processed.
[0060] Wherein, the third acquisition module includes:
[0061] A splicing processing unit, used for splicing the global color features and the image semantic features to obtain the target image features;
[0062] The first output unit is used to input the target image feature into the target generation model, and output the enhancement parameters of the target image feature in M image information dimensions respectively through the target generation model.
[0063] Among them, the enhancement parameters on the M image information dimensions include brightness enhancement parameters, contrast enhancement parameters and saturation enhancement parameters;
[0064] The first output unit is specifically used for:
[0065] Inputting the target image features into the target generation model, performing weighted processing on the target image features according to the weight matrix corresponding to the first regression network in the target generation model, and obtaining the brightness enhancement parameters corresponding to the video frame to be processed;
[0066] According to the weight matrix corresponding to the second regression network in the target generation model, the target image features are weighted to obtain the contrast enhancement parameters corresponding to the video frame to be processed;
[0067] According to the weight matrix corresponding to the third regression network in the target generation model, the target image features are weighted to obtain the saturation enhancement parameters corresponding to the video frame to be processed.
[0068] The target video frame set includes video frames T j ; j is a positive integer less than or equal to the number of video frames included in the target video frame set;
[0069] Image enhancement parameters include:
[0070] The first determining unit is used to determine the brightness enhancement parameter and the video frame T j The product of is determined as the video frame T j a corresponding first candidate video frame;
[0071] The second determining unit is used to obtain the global average pixel value corresponding to the first candidate video frame, obtain a first difference between the pixel value included in the first candidate video frame and the global average pixel value, and determine the video frame T according to the global average pixel value and the product of the first difference and the contrast enhancement coefficient. j a corresponding second candidate video frame;
[0072] The third determining unit is used to obtain the channel average pixel value corresponding to the second candidate video frame, obtain the second difference between the pixel value included in the second candidate video frame and the channel average pixel value, and determine the video frame T according to the channel average pixel value and the product of the second difference and the saturation enhancement coefficient. j The corresponding target image data.
[0073] The third acquisition module further includes:
[0074] A fourth acquisition unit, used to acquire sample video frames and label information corresponding to the sample video frames;
[0075] A second output unit is used to input the sample global color features and the sample image semantic features corresponding to the sample video frame into the initial generation model, and output the prediction enhancement parameters corresponding to the sample video frame through the initial generation model;
[0076] A fourth determining unit, configured to determine a loss function corresponding to the initial generation model according to label information corresponding to the sample video frame and a prediction enhancement parameter corresponding to the sample video frame;
[0077] The fifth determination unit is used to iteratively train the initial generation model according to the loss function, and when the initial generation model meets the convergence condition, determine the initial generation model that meets the convergence condition as the target generation model.
[0078] In one aspect, the present application provides a computer device, including: a processor and a memory;
[0079] The memory is used to store the computer program, and the processor is used to call the computer program to perform the following steps:
[0080] A target video frame set is obtained from the video data to be processed; the video frames included in the target video frame set have the same attribute information;
[0081] Obtain a video frame to be processed from the target video frame set, obtain a global color feature corresponding to the video frame to be processed, and obtain an image semantic feature corresponding to the video frame to be processed;
[0082] According to the global color features and image semantic features, the enhancement parameters of the video frame to be processed in M image information dimensions are obtained; M is a positive integer;
[0083] According to the M enhancement parameters, image enhancement is performed on the video frames included in the target video frame set to obtain target image data corresponding to each video frame in the target video frame set.
[0084] In one aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the computer program performs the following steps:
[0085] A target video frame set is obtained from the video data to be processed; the video frames included in the target video frame set have the same attribute information;
[0086] Obtain a video frame to be processed from the target video frame set, obtain a global color feature corresponding to the video frame to be processed, and obtain an image semantic feature corresponding to the video frame to be processed;
[0087] According to the global color features and image semantic features, the enhancement parameters of the video frame to be processed in M image information dimensions are obtained; M is a positive integer;
[0088] According to the M enhancement parameters, image enhancement is performed on the video frames included in the target video frame set to obtain target image data corresponding to each video frame in the target video frame set.
[0089] In one aspect, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method in the above aspect.
[0090] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in the above aspect of the embodiment of the present application.
[0091] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor, so that a computer device with a processor executes the method provided in the above aspect of the embodiment of the present application.
[0092] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the above aspect.
[0093] In the embodiment of the present application, a target video frame set is obtained from the video data to be processed, and the video frames included in the target video frame set have the same attribute information, and the video frames to be processed are obtained from the target video frame set, and the global color features corresponding to the video frames to be processed are obtained, and the image semantic features corresponding to the video frames to be processed are obtained. According to the global color features and the image semantic features, the enhancement parameters of the video frames to be processed in M image information dimensions are obtained, and according to the M enhancement parameters, the video frames included in the target video frame set are image enhanced to obtain the target image data corresponding to each video frame in the target video frame set. In the present application, the target video frame is obtained from one or more video frame sets of the video data to be processed, and the video frame to be processed is obtained from the target video frame set, and the enhancement coefficients of the video frames to be processed in M image information dimensions are obtained according to the global color features and the image semantic features corresponding to the video frames to be processed, and the enhancement coefficients are used as the enhancement coefficients corresponding to the target video frame set, and the video frames included in the target video frame set are image enhanced to obtain the target image data corresponding to each video frame in the target video frame set. It can be seen that it is only necessary to obtain the enhancement parameters for one video frame in a video frame set, so as to determine the enhancement parameters corresponding to the video frame as the enhancement parameters of the corresponding video frame set, which can reduce the amount of calculation and improve the efficiency of image enhancement. At the same time, video frames with the same attributes can use the same set of enhancement parameters, which can reduce the risk of color jumps in the image, thereby improving the enhancement quality of the video data to be processed. BRIEF DESCRIPTION OF THE DRAWINGS
[0094] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0095] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;
[0096] Figure 2 is a schematic diagram of a scene graph for image enhancement provided in an embodiment of the present application;
[0097] Figure 3 is a timing diagram of an image processing method provided in an embodiment of the present application;
[0098] Figure 4 is a timing diagram of an image processing method provided in an embodiment of the present application;
[0099] Figure 5It is a schematic diagram of the structure of a regression network in a target generation model provided in an embodiment of the present application;
[0100] Figure 6 is a schematic diagram of a target generation model provided in an embodiment of the present application;
[0101] Figure 7 is a schematic diagram of prediction enhancement parameters corresponding to a target generation model provided in an embodiment of the present application;
[0102] Figure 8 is a schematic diagram of a comparison between a video frame to be processed and corresponding target image data provided by an embodiment of the present application;
[0103] Fig. 9 is a schematic diagram of a video image processing scenario provided by an embodiment of the present application;
[0104] Fig.10 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;
[0105] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0106] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0107] This application relates to artificial intelligence (AI) technology, blockchain (BlockChain) technology and cloud technology. Artificial intelligence is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large image processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application specifically relates to computer vision technology (Computer Vision, CV) under artificial intelligence technology.
[0108] Among them, computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, locate and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The present application specifically relates to image processing under computer vision. By obtaining the enhancement parameters corresponding to the video frame to be processed, the image enhancement is performed on the video data to be processed according to the enhancement parameters, and the target image data corresponding to the video data to be processed is obtained, which can improve the enhancement quality corresponding to the video data to be processed.
[0109] The present application relates to artificial intelligence cloud services under cloud technology. Artificial intelligence cloud services refer to AIaaS (AI as aService, which means "AI as a Service" in Chinese). This is a mainstream service mode of artificial intelligence platforms. Specifically, the AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through API interfaces. Some senior developers can also use the AI framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services. In the present application, one or more image enhancement cloud services provided by the platform can be accessed through API interfaces. Through the one or more image enhancement cloud services, image enhancement is performed on the video data to be processed, and the target image data corresponding to the video data to be processed is obtained, which can improve the enhancement quality of the video data to be processed.
[0110] The present application relates to cloud storage under cloud technology. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world. At present, the storage method of the storage system is: create a logical volume. When creating a logical volume, a physical storage space is allocated to each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, the data is stored on the file system. The file system divides the data into many parts, each of which is an object. The object not only contains data but also contains additional information such as data identification (ID entity, ID). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.
[0111] The process of allocating physical storage space for a logical volume by a storage system is specifically as follows: according to the estimated capacity of the object stored in the logical volume (the estimated capacity often has a large margin relative to the actual capacity of the object to be stored) and the group of the Redundant Array of Independent Disk (RAID), the physical storage space is pre-divided into stripes, and one logical volume can be understood as a stripe, thereby allocating physical storage space for the logical volume. In the present application, the to-be-processed video data corresponding to different clients can be stored in different logical volumes, that is, the to-be-processed video data of different clients can all be stored in the file system; for the to-be-processed video data corresponding to any client, the file system can divide it into multiple parts, that is, the to-be-processed video data is divided into multiple video frame sets, each part is an object, and the object can include not only the video frame, but also the frame code corresponding to the video frame, the shooting time or rendering time corresponding to the video frame, etc. The file system can write each object into the physical storage space of the logical volume, and the file system will record the storage location information of each object, so that when performing image processing on each video frame, the video frame in each video frame set can be accessed according to the storage location information.
[0112] Blockchain is a new application mode of computer technology such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. It is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer and application service layer; blockchain can be composed of multiple serial transaction records (also known as blocks) that are connected and protected by cryptography. The distributed ledger connected by blockchain allows multiple parties to effectively record transactions and can permanently check this transaction (not tamperable). Among them, the consensus mechanism refers to the mathematical algorithm that realizes the establishment of trust and the acquisition of rights and interests between different nodes in the blockchain network; that is, the consensus mechanism is a mathematical algorithm recognized by all network nodes of the blockchain. In this application, the video data to be processed and the target image data corresponding to the video data to be processed can be stored in the blockchain to prevent the video data to be processed and the target image data corresponding to the video data to be processed from being illegally tampered with, thereby improving the security of the data.
[0113] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a server 10d and a user terminal cluster, and the user terminal cluster may include one or more user terminals, and the number of user terminals is not limited here. Figure 1 As shown, the user terminal cluster may specifically include user terminal 10a, user terminal 10b and user terminal 10c, etc. Among them, server 10d may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. User terminal 10a, user terminal 10b and user terminal 10c, etc. may include: smart phones, tablet computers, laptop computers, PDAs, mobile Internet devices (mobile internet devices, MIDs), wearable devices (such as smart watches, smart bracelets, etc.) and smart TVs, etc., which have video data processing functions. As Figure 1 As shown, the user terminal 10a, the user terminal 10b and the user terminal 10c can respectively establish a network connection with the server 10d, so that each user terminal can exchange data with the server 10d through the network connection.
[0114] For ease of understanding, the embodiments of the present application can be Figure 1 Select a user terminal from the user terminal cluster shown as the target user terminal. Figure 1 The user terminal 100a shown is used as a target user terminal, and an application client with an image processing function can be integrated in the target user terminal. For example, the user terminal 100a can obtain video data, such as small video data, live broadcast data, etc., through a camera device, and send the video data to the server 10. After receiving the video data sent by the target user terminal 100a, the server 10 can use the video data sent by the target user terminal 100a as the video data to be processed, and obtain the global color features and image semantic features corresponding to the video frame to be processed in the video data to be processed, and obtain the enhancement parameters corresponding to the video frame to be processed according to the global color features and image semantic features, and perform image enhancement on the video frame to be processed according to the enhancement parameters to obtain the target image data corresponding to the video frame to be processed. In this way, the color of the video frame in the video data to be processed is enhanced to improve the display effect of the video frame, improve the user's viewing experience, and at the same time, the efficiency of image enhancement can be improved.
[0115] like Figure 2 As shown, Figure 2 is an image enhancement scene graph provided by an embodiment of the present application, such as Figure 2 The terminal 1 shown can be the above Figure 1 The client with image enhancement authority integrated in the user terminal 10a shown in the figure, the terminal 2 can be the above Figure 1 The client of the image enhancement authority integrated in the user terminal 10b shown in the figure may be the above-mentioned service device. Figure 1 The server 10d shown. Figure 2 As shown, user 1 can live broadcast through terminal 1, and terminal 1 can shoot the user's behavior through its configured camera device, obtain video data corresponding to user 1, and send the video data corresponding to user 1 to the server. The server can receive the video data corresponding to user 1 sent by terminal 1, determine the video data corresponding to user 1 as the video data to be processed, divide and process the video data to be processed, and obtain N video frame sets, where N is a positive integer, such as N can be 1, 2, 3, ... For ease of understanding, taking the target video frame set in the N video frame sets as an example, the target video frame set is any one of the N video frame sets, the server can obtain the global color features and image semantic features corresponding to the video frame to be processed in the target video frame set, and obtain the enhancement parameters of the video frame to be processed according to the global color features and image semantic features, and then perform image enhancement on the video frame to be processed according to the enhancement parameters to obtain the target image data corresponding to the video frame to be processed, so as to improve the image quality corresponding to the video frame to be processed. At the same time, the server can also use the enhancement parameter of the video frame to be processed as the enhancement coefficient corresponding to the target video frame set, and perform image enhancement on the video frames in the target video frame set according to the enhancement coefficient to obtain the target image data corresponding to the target video frame set, thereby obtaining the target image data corresponding to the video data to be processed. After the server obtains the target image data corresponding to the video data to be processed, it can send the target image data to terminal 2, and user 2 can watch the target image data corresponding to user 1 in terminal 2. Figure 2 As shown, after the terminal 1 obtains the video data corresponding to the user 1, since the user 1 is affected by factors such as light and balance during the shooting process, the display effect corresponding to the captured video data is poor (such as dim light or overexposure, etc.), resulting in a poor viewing experience for the viewing user. Therefore, the server performs image enhancement on the video data of user 1, and sends the enhanced target video data to the user who wants to watch it (such as user 2), which can improve the enhancement quality of the video data, thereby improving the user's viewing experience. In addition, the present application can also improve the efficiency of image enhancement.
[0116] See also Figure 3 , Figure 3 FIG. 1 is a timing diagram of an image processing method provided in an embodiment of the present application. It can be understood that the image processing method can be interactively executed by a client and a service device, and the client can be the above-mentioned Figure 1 The client integrated in any user terminal in the user terminal cluster shown in the figure may be a separate server (for example, the above Figure 1 The server 10d) shown in the figure may be a server cluster composed of multiple servers, or a user terminal, etc. Figure 3 As shown, the image processing method may include the following steps:
[0117] Step S101, obtaining a target video frame set from the video data to be processed.
[0118] Specifically, due to improper operation when shooting videos, the video data obtained may inevitably be overexposed or dimly lit. Therefore, after the user shoots the user video data (such as beauty videos, short life videos, etc.), the user video data obtained can be processed by image processing (such as image enhancement) to improve the display effect corresponding to the user video data. In an embodiment of the present application, after the computer device obtains the video data to be processed (such as receiving the video data uploaded by the user, and determining the video data uploaded by the user as the video data to be processed), it can obtain a target video frame set in the video data to be processed. The target video frame set is any one of the one or more video frame sets in the video data to be processed. The video frames contained in the target video frame set have the same attribute information, such as the video frames in the same video frame set belong to the same scene, that is, the scene pictures corresponding to the video frames in the same video frame set are similar.
[0119] Optionally, a specific method for the computer device to obtain the target video frame set from the video data to be processed may include: obtaining the video data to be processed, dividing the video data to be processed into L video frames, and obtaining color histograms corresponding to the L video frames, where L is a positive integer, such as L can be 1, 2, 3, ...; the L video frames include video frames T having an adjacent relationship i and video frame T i+1 ; According to the video frame T i and video frame T i+1 The corresponding color histograms are obtained to obtain the video frame T i and video frame T i+1 similarity distance between them; i is a positive integer less than L, such as i can be 1, 2, 3, ...; L video frames are divided according to the similarity distance to obtain N video frame sets; the similarity distance between adjacent video frames in each video frame set is less than the distance threshold, N is a positive integer, such as N can be 1, 2, 3, ...; any one video frame set is selected from the N video frame sets as the target video frame set.
[0120] Specifically, after the computer device obtains the video data to be processed, it can divide the video data to be processed into L video frames, and obtain the color histograms corresponding to the L video frames. For ease of understanding, the video frames T with adjacent relationships among the L video frames are taken as follows: i and video frame T i+1 For example, the computer device obtains the video frame T i and video frame T i+1 After obtaining the corresponding color histograms, we can compare the video frame T i and video frame T i+1 The corresponding color histograms are obtained to obtain the video frame T i and video frame T i+1 The similarity distance between the two video frames is calculated by dividing the L video frames in the video data to be processed into N video frame sets, wherein the similarity distance between adjacent video frames in each video frame set is less than the distance threshold, and any one of the N video frame sets is selected as the target video frame set.
[0121] For example, L video frames include four video frames T1, T2, T3, and T4, which have a continuous shooting time relationship. The color histograms corresponding to the video frames T1, T2, T3, and T4 can be obtained, and the similarity distances of adjacent video frame times can be calculated according to the sequence of shooting time and the color histograms of each video frame. For example, the similarity distance between video frame T1 and video frame T2 is calculated according to the color histograms corresponding to video frame T1 and video frame T2. If the similarity distance between video frame T1 and video frame T2 is greater than or equal to the distance threshold, video frame T1 and video frame T2 are divided into different video frame sets; if the similarity distance between video frame T1 and video frame T2 is less than the distance threshold, video frame T1 and video frame T2 are divided into the same video frame set. After dividing video frame T1 and video frame T2, the distance threshold between video frame T2 and video frame T3 can be calculated according to the color histograms corresponding to video frame T2 and video frame T3 respectively. Similarly, if the similarity distance between video frame T2 and video frame T3 is greater than or equal to the distance threshold, video frame T2 and video frame T3 are divided into different video frame sets; if the similarity distance between video frame T2 and video frame T3 is less than the distance threshold, video frame T2 and video frame T3 are divided into the same video frame set. When the similarity distance between any adjacent video frames among video frame T1, video frame T2, video frame T3 and video frame T4 is less than the distance threshold, video frame T1, video frame T2, video frame T3 and video frame T4 can be divided into the same video frame set. In this way, L video frames are divided to obtain N video frame sets. Dividing the video data to be processed into N video frame sets and then performing image processing on the video frames in each video frame set can avoid obvious color jumps and improve the effect of image enhancement, thereby improving the user's viewing experience.
[0122] Optionally, the specific method for the computer device to obtain the color histograms corresponding to the L video frames may include: dividing the color space into P image color intervals, obtaining the video frame T in the L video frames i ; P is a positive integer, such as P can be 1, 2, 3, ...; according to the video frame T i The image color space where the pixel points in the image are located is counted, and the number of pixels contained in each image color interval is counted; according to the number of pixels corresponding to each of the P image color intervals, a video frame T is generated. i The corresponding color histogram.
[0123] Specifically, the computer device can divide the color space into P image color intervals, the color histogram includes multiple small rectangles (bins), each image color interval is a small rectangle in the color histogram, and the height of each small rectangle is used to indicate the number of pixels in the image that are in the image color interval corresponding to the small rectangle. i , according to the video frame T i The image color space where the pixels in the image are located is counted, and the number of pixels contained in each image color interval is counted, that is, the video frame T i The pixels in the image are divided into various image color intervals, and the video frame T is calculated. i According to the number of pixels corresponding to the P image color intervals, the height of the small rectangle corresponding to the P image color intervals is determined to generate the video frame T i The corresponding color histogram is obtained to obtain the color histograms corresponding to the L video frames. The color histogram can be a histogram created based on the HSV color space (HueSaturation Value), Luv color space (CIELUV) or Lab color space (CIELab). The HSV color space is a color space created based on the intuitive characteristics of color. The parameters of the HSV color space are: hue (H), saturation (S), and brightness (V). In the Luv color space, L represents the brightness of the object, and u and v are chromaticity, which are obtained by a simple transformation of the image space and have visual unity. The Lab color space uses a digital method to describe human visual perception. The L component in the Lab color space is used to represent the brightness of the pixel, and the value range is [0,100], which means from pure black to pure white. a represents the range from red to green, and the value range is [127, -128]. b represents the range from yellow to blue, and the value range is [127, -128]. For example, the present application may generate color histograms corresponding to L video frames respectively based on the Lab color space.
[0124] Step S102, obtaining a to-be-processed video frame in the target video frame set, obtaining a global color feature corresponding to the to-be-processed video frame, and obtaining an image semantic feature corresponding to the to-be-processed video frame.
[0125] Specifically, the computer device can obtain a video frame to be processed from the target video frame set, and the video frame to be processed is any video frame in the target video frame set, that is, a video frame is randomly selected from the target video frame set as the video frame to be processed. The computer device can obtain the global color feature corresponding to the video frame to be processed, and the global color feature can refer to the feature that can represent the entire image and is used to describe the overall features such as the color and shape of the image or target, and obtain the image semantic feature corresponding to the video frame to be processed.
[0126] Optionally, the computer device obtains the global color features corresponding to the video frame to be processed, and the specific method of obtaining the image semantic features corresponding to the video frame to be processed may include: resizing the video frame to be processed to obtain a candidate video frame with a target size, and obtaining the global color features corresponding to the video frame to be processed according to the color histogram corresponding to the candidate video frame. The candidate video frame is input into the feature extraction model, and the candidate video frame is convolved according to the convolution layer in the feature extraction model to obtain the image convolution features corresponding to the candidate video frame, and the image convolution features are residually operated according to the residual layer in the feature extraction model to obtain the image semantic features corresponding to the video frame to be processed. Among them, the color histogram corresponding to the candidate video frame can also be any one of the histograms obtained based on the HSV color space, the Luv color space or the Lab color space, that is, the color histogram corresponding to the candidate video frame can be a histogram obtained based on the HSV color space, or a histogram obtained based on the Luv color space, or a histogram obtained based on the Lab color space, and the embodiment of the present application is not limited here.
[0127] Specifically, the computer device can first resize the video frame to be processed, such as adjusting the size of the video frame to be processed to a small size, such as 224*224, so as to improve the operation efficiency when extracting features from the video frame to be processed. After the computer device obtains the candidate video frame with the target size, it can obtain the color histogram corresponding to the candidate video frame, such as obtaining the color histogram corresponding to the candidate video frame based on the Lab color space, and extract the global color features corresponding to the candidate video frame according to the color histogram corresponding to the candidate video frame. Among them, when extracting the global color features corresponding to the candidate video frame according to the color histogram corresponding to the candidate video frame, the color feature extraction network can be used to extract features from the color histogram corresponding to the candidate video frame. Each channel in the color feature extraction network can have 20 small rectangles (bins), with a total of 20*20*20=8000 dimensions. Optionally, when obtaining the color histogram corresponding to the candidate video frame, the computer device can obtain the color histogram corresponding to the candidate video frame according to the above method for obtaining the color histogram corresponding to L video frames in the video data to be processed, and obtain the global color features corresponding to the video frame to be processed according to the color histogram. Optionally, since the color histogram corresponding to each video frame in the video data to be processed has been obtained when the video data to be processed is divided to obtain a set of N video frames, the video to be processed also belongs to the video data to be processed. Therefore, when the video data to be processed is divided, the color histogram corresponding to each video frame in the video data to be processed can be stored according to the video frame identifier. In this way, when obtaining the color histogram corresponding to the candidate video frame, the color histogram corresponding to the corresponding video frame can be directly obtained from the corresponding database as the color histogram corresponding to the candidate video frame.
[0128] Specifically, when the computer device obtains the image semantic features corresponding to the video frame to be processed, the candidate video frame can be input into the feature extraction model. The feature extraction model can be a lightweight convolutional neural network. The lightweight convolutional neural network can reduce the number of parameters, thereby reducing the amount of calculation. The computer device performs a convolution operation on the candidate video frame according to the convolution layer in the feature extraction model to obtain the image convolution features corresponding to the candidate video frame, and performs a residual operation on the image convolution features corresponding to the candidate video frame according to the residual layer in the feature extraction model to obtain the image semantic features corresponding to the video frame to be processed. Among them, the feature extraction model can be MobileNetV1 (lightweight model), MobileNetV2 (lightweight model), ResNet (a residual network), etc. The embodiment of the present application takes MobileNetV2 as an example to describe in detail the feature extraction process of the video frame to be processed. MobileNetV2 uses a depth-level separable convolution 3x3conv (convolution), which has a much smaller computational complexity than standard convolution. Therefore, in order to extract more features, MobileNetV2 first uses 1x1conv to increase the number of channels, and then uses 1x1conv to reduce the number of channels, forming a module with small ends and a large middle (i.e., inverted residual module), which is the opposite of Residual Block. When using MobileNetV2 to obtain the image semantic features corresponding to the video frame to be processed, the output image features of the previous layer of the fully connected classification layer in MobileNetV2 can be taken as the image semantic features corresponding to the video frame to be processed, that is, if the previous layer of the fully connected classification layer in MobileNetV2 is an inverted residual layer V, the output image features of the inverted residual layer V are determined as the image semantic features corresponding to the video frame to be processed. When MobileNetV2 is used to extract image semantic features from the video frames to be processed, the inverted residual module mentioned above can be used in the MobileNetV2 network structure to extract features from the video frames to be processed. This can reduce the computer and obtain the image semantic features corresponding to the video frames to be processed with less calculation amount, thereby improving the efficiency of feature extraction.
[0129] Step S103, obtaining enhancement parameters of the to-be-processed video frame in M image information dimensions respectively according to the global color features and the image semantic features.
[0130] Specifically, after the computer device obtains the global color features and image semantic features corresponding to the video frame to be processed, it can perform feature fusion extraction on the global color features and image semantic features, determine the image attribute information such as image chromaticity information, image brightness information, image contrast information and image saturation information corresponding to the video frame to be processed, and determine whether the image attribute information is within the normal threshold. If the image attribute feature is not within the normal threshold, the difference between the image attribute information corresponding to the video frame to be processed and the normal threshold is obtained, and the enhancement parameters corresponding to the video frame to be processed are determined according to the difference, such as determining the enhancement parameters of the video frame to be processed in M image information dimensions, and the image information dimensions may include brightness information dimension, contrast information dimension and saturation information dimension. According to the enhancement parameters, the video frame to be processed is adjusted, and the image attribute information that is not within the normal threshold is adjusted so that the adjusted image attribute information is within the normal threshold. If the image brightness information exceeds the normal brightness range (such as overexposure), the image brightness can be reduced according to the corresponding enhancement parameters; if the image brightness information is lower than the normal brightness range (such as too dark light), the image brightness can be increased according to the corresponding enhancement parameters.
[0131] Step S104: performing image enhancement on the video frames included in the target video frame set according to the M enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set.
[0132] Specifically, after the computer device obtains the enhancement parameters of the to-be-processed video frame in M image information dimensions, the video frames included in the target video frame set can be enhanced according to the M enhancement parameters to obtain the target image data corresponding to each video frame in the target video frame set, where M is a positive integer, such as M can be 1, 2, 3, .... For example, the M enhancement parameters can include a brightness enhancement parameter, a contrast enhancement parameter, and a saturation enhancement parameter, so each video frame in the target video frame set can be enhanced according to the brightness enhancement parameter, the contrast enhancement parameter, and the saturation enhancement parameter to obtain the target image data corresponding to each video frame in the target video frame set. For example, take the video frame i in the target video frame set as an example for explanation, i is a positive integer, such as i can be 1, 2, 3, ..., the video frame i refers to any video frame in the target video frame set, and according to the brightness enhancement parameter, the brightness of the video frame i is adjusted to obtain the video frame i after brightness enhancement, and at this time, the brightness of the video frame i after brightness enhancement is within the normal threshold range. According to the contrast enhancement parameter, the contrast of the video frame i after brightness enhancement is adjusted to obtain the contrast-adjusted video frame i, at which time the brightness and contrast corresponding to the contrast-adjusted video frame i are within the normal threshold range. According to the saturation enhancement parameter, the saturation of the video frame i after contrast adjustment is adjusted to obtain the saturation-adjusted video frame i, at which time the brightness, contrast and saturation corresponding to the video frame I are within the normal threshold range, and the saturation-adjusted video frame i is used to determine the target image data corresponding to the video frame i.
[0133] Among them, since the video frames in each video frame set in the video data to be processed have the same attribute information (such as the video frames in each video frame set belong to the same scene), a video frame can be randomly selected from the target video frame set as the video frame to be processed. After the enhancement parameters of the video frame to be processed are extracted and the enhancement parameters of the video frame to be processed in M image information dimensions are obtained, the enhancement parameters of the video frame to be processed in M image information dimensions can be determined as the enhancement parameters of the target video frame set in M image information dimensions. Each video frame in the target video frame set can be image enhanced according to the enhancement parameters in M image information dimensions to obtain the target image data corresponding to each video frame in the target video frame set, thereby obtaining the enhancement parameters corresponding to each video frame set in the video data to be processed, and then obtaining the target image data corresponding to each video frame in the video frame to be processed according to the enhancement parameters corresponding to each video frame set. In this way, the video frames in the same video frame set use the same set of enhancement parameters, which can not only avoid visible color jumps between video frames in the same scene, but also only require one enhancement parameter prediction for a video frame set, which can greatly reduce the amount of computation in video applications and improve image processing efficiency.
[0134] Optionally, after the computer device obtains the enhancement coefficient corresponding to each video frame set in the video data to be processed, the computer device can perform threshold processing on the enhancement parameters corresponding to each video frame set, so that the enhancement parameters corresponding to each video frame do not exceed the enhancement threshold, and different video frame sets or different application scenarios correspond to different enhancement thresholds. In this way, in order to avoid serious color jumping between video frames, the display effect of the target image data can be improved, thereby improving the viewing experience of the viewing user. Among them, the enhancement threshold corresponding to the enhancement parameter can be defined according to the specific situation, and the embodiment of the present application is not limited here. In addition, the embodiment of the present application can also use the LUT (Look Up Table) lookup algorithm for optimization when performing image enhancement on each video frame in the video data to be processed. The LUT lookup algorithm can write data into RAM (Random Access Memory, which is an internal memory that directly exchanges data with the CPU) in advance. Whenever a signal is input, it is equivalent to inputting an address to look up the table, find out the content corresponding to the address, and then output it. In fact, it is a mapping table of pixel grayscale values. The actual sampled pixel grayscale value is transformed (such as threshold transformation, inversion, binarization, contrast adjustment, etc.), and another corresponding grayscale value is programmed, so that the useful information of the image can be highlighted, thereby enhancing the image. Since the pixel value of each channel in the image has only 255 integer values, the 255 values corresponding to the image can be calculated in advance and stored in the default threshold table. In this way, when enhancing the video frame, you can directly search quickly in the threshold table, which is much more efficient than performing the same repeated calculation on each pixel point, which can improve the efficiency of image enhancement.
[0135] In the embodiment of the present application, a target video frame set can be obtained from the video data to be processed, wherein the video frames included in the target video frame set have the same attribute information, the video frames to be processed are obtained from the target video frame set, the global color features corresponding to the video frames to be processed are obtained, and the image semantic features corresponding to the video frames to be processed are obtained. According to the global color features and the image semantic features, the enhancement parameters of the video frames to be processed in M image information dimensions are obtained, and according to the M enhancement parameters, the video frames included in the target video frame set are image enhanced to obtain the target image data corresponding to each video frame in the target video frame set. In the present application, the video data to be processed is divided according to the color histogram corresponding to each video frame to obtain N video frame sets, wherein the video frames included in each video frame set have the same attribute information, so that the consistency of the division of the video data to be processed can be guaranteed, that is, the video frames included in each video frame set are guaranteed to have the same attribute information (such as the video frames in the same video frame set all belong to the same scene screen). In this way, when acquiring enhancement parameters for each video frame set, a video frame can be directly randomly acquired from each video frame set as the video frame to be processed, and the enhancement coefficients of the video frame to be processed in M image information dimensions are acquired according to the global color features and image semantic features corresponding to the video frame to be processed, and the enhancement coefficients are used as the enhancement coefficients corresponding to the corresponding video frame set, and the video frames included in the video frame set are respectively enhanced to obtain the target image data corresponding to each video frame in each video frame set. It can be seen that it is only necessary to acquire enhancement parameters for one video frame in a video frame set, so as to determine the enhancement parameters corresponding to the video frame as the enhancement parameters of the corresponding video frame set, which can reduce the amount of calculation and improve the efficiency of image enhancement. At the same time, video frames with the same attributes can use the same set of enhancement parameters, which can reduce the risk of color jump phenomenon in the image, and thus improve the enhancement quality of the video data to be processed.
[0136] See also Figure 4 , Figure 4 FIG. 1 is a timing diagram of an image processing method provided in an embodiment of the present application. It can be understood that the image processing method can be interactively executed by a client and a service device, and the client can be the above-mentioned Figure 1 The client integrated in any user terminal in the user terminal cluster shown in the figure may be a separate server (for example, Figure 1 The server 10d) shown in the figure may be a server cluster composed of multiple servers, or a user terminal, etc. Figure 4 As shown, the image processing method may include the following steps:
[0137] Step S201 , obtaining a target video frame set from the video data to be processed; the video frames included in the target video frame set have the same attribute information.
[0138] Step S202, obtaining a to-be-processed video frame in the target video frame set, obtaining a global color feature corresponding to the to-be-processed video frame, and obtaining an image semantic feature corresponding to the to-be-processed video frame.
[0139] The specific contents of step S201-step S202 can be found in Figure 3 The contents of step S101 to step S102 in the embodiment of the present application will not be repeated here.
[0140] Step S203, concatenating the global color features and the image semantic features to obtain target image features.
[0141] Specifically, after the computer device obtains the global color features and image semantic features corresponding to the video frame to be processed, the global color features and image semantic features can be spliced to obtain the target image features. For example, the global color features extracted by the Lab color histogram have a total of 8000 dimensions, and the image semantic features extracted by mobileNetV2 have 1280 dimensions. After splicing the global color features and image semantic features, a 9280-dimensional target image feature can be generated.
[0142] Step S204, inputting the target image features into the target generation model, and outputting the enhancement parameters of the target image features in M image information dimensions respectively through the target generation model; M is a positive integer.
[0143] Specifically, after the computer device obtains the target image features corresponding to the video frame to be processed, the target image features can be input into the target generation model, and the target image is feature extracted through the target generation model, and the enhancement parameters of the M image information dimensions of the video frame to be processed are output. The target generation model may include M regression networks, which may be fully connected regression networks. The number of regression networks in the target generation model may be determined by the number of M image information dimensions to be predicted. One image information dimension corresponds to one regression network, that is, one regression network is used to predict the enhancement parameters of one image information dimension.
[0144] Optionally, the enhancement parameters on the M image information dimensions include brightness enhancement parameters, contrast enhancement parameters, and saturation enhancement parameters. The specific method in which the computer device outputs the enhancement parameters of the target image features on the M image information dimensions through the target generation model may include: inputting the target image features into the target generation model, performing weighted processing on the target image features according to the weight matrix corresponding to the first regression network in the target generation model, and obtaining the brightness enhancement parameters corresponding to the video frame to be processed. Performing weighted processing on the target image features according to the weight matrix corresponding to the second regression network in the target generation model, and obtaining the contrast enhancement parameters corresponding to the video frame to be processed. Performing weighted processing on the target image features according to the weight matrix corresponding to the third regression network in the target generation model, and obtaining the saturation enhancement parameters corresponding to the video frame to be processed.
[0145] Specifically, each regression network in the target generation model may include one or more fully connected layers, and an activation function layer may be connected after the fully connected layer. The activation function layer may be used to perform nonlinear combination of the output features of the fully connected layer and enhance the output features of the fully connected layer output. When the regression network includes a fully connected layer, an activation function layer may be connected after the fully connected layer. At this time, the parameters of the fully connected layer and the activation function layer may be referred to as the weight matrix corresponding to the regression network, and the output result of the activation function layer may be used as the output result of the regression network; when the regression network includes multiple fully connected layers, the fully connected layers and the activation function layers may be alternately connected, that is, one fully connected layer may be connected after an activation function layer. Optionally, the last fully connected layer may be used as the output layer of the regression network, that is, the last fully connected layer of the regression network does not need to be connected after the activation function layer, and the parameters of all fully connected layers and activation function layers contained in the regression network may be referred to as the weight matrix of the regression network. The computer device may input the target image features into the target generation model, and perform weighted processing on the target image features according to the weight matrix corresponding to the first regression network in the target generation model to obtain the brightness enhancement parameters corresponding to the video frame to be processed. According to the weight matrix corresponding to the second regression network in the target generation model, the target image features are weighted to obtain the contrast enhancement parameters corresponding to the video frame to be processed. According to the weight matrix corresponding to the third regression network in the target generation model, the target image features are weighted to obtain the saturation enhancement parameters corresponding to the video frame to be processed. The M regression networks included in the target generation model can be used to predict the enhancement parameters on different image information dimensions respectively. The M regression networks can have the same network structure, but the weight matrices (network parameters) corresponding to the M regression networks are different.
[0146] like Figure 5 As shown, Figure 5is a schematic diagram of the structure of a regression network in a target generation model provided in an embodiment of the present application, such as Figure 5 As shown, since there is a correlation between the brightness, contrast and saturation of the image, the target generation model can be used to predict the enhancement parameters of the video frame to be processed in the contrast information dimension, the brightness information dimension and the saturation information dimension respectively. In this way, the target generation model can include regression networks corresponding to the contrast information dimension, the brightness information dimension and the saturation information dimension respectively, that is, the first regression network can be used to predict the enhancement parameters of the video frame to be processed in the brightness information dimension; the second regression network is used to predict the enhancement parameters of the video frame to be processed in the contrast information dimension; the third regression network is used to predict the enhancement parameters of the video frame to be processed in the saturation information dimension. Optionally, assuming that the first regression network, the second regression network and the third regression network can all include four fully connected layers and three activation function layers (the last fully connected layer can be used as the output layer of the regression network), the specific structure content of each regression network can be as follows Figure 5 As shown, each regression network may include a first fully connected layer, a second fully connected layer, and a third fully connected layer. Figure 5 As shown in the figure, the output dimension of the first fully connected layer is 4096, and the computing unit is fully connected 9280*4096. After the first fully connected layer, there is also a first activation layer, namely, ReLU (Rectified Linear Unit, linear rectification function, also known as rectified linear unit, a commonly used activation function in artificial neural networks) activation layer. The output dimension of the ReLU activation layer is 4096. The image features output by the first fully connected network layer are activated by the ReLU activation function in the ReLU activation layer to enhance the image features output by the first fully connected layer. The output dimension of the second fully connected layer is 4096, and the computing unit is fully connected 4096*4096. After the second fully connected layer, there is also a second activation layer (i.e., the second activation layer), and the function of the second activation layer is the same as that of the ReLU activation layer in the first fully connected layer. The output dimension of the third fully connected layer is 1024, and the computing unit is 4096*1024. Similarly, after the third fully connected layer, there is also a third activation layer, and the function is the same as that of the ReLU activation layer in the first fully connected network. Finally, each regression network also includes an output layer, which uses a linear function to output the enhanced parameters obtained in the corresponding regression network. The output layer can also be a fully connected layer. Figure 5 As shown, the output layer in the target generation model may refer to the fourth fully connected layer, the output dimension is 1, and the corresponding computing unit is 1024*1, which is used to output the enhanced parameters predicted in the corresponding regression network.
[0147] For example, taking the brightness enhancement parameter corresponding to the video frame to be processed obtained by the first regression network as an example, the enhancement parameters on each dimension of the image information obtained by the regression network in the target generation model are described in detail. The computer device can input the target image feature into the first regression network, and perform weighted processing on the target image feature through the weight matrix corresponding to the first fully connected layer in the first regression network to obtain the first output feature corresponding to the target image feature in the first fully connected layer. The first output feature is activated through the first activation layer after the first fully connected layer to obtain the first output feature after activation. The first output feature after activation is weighted through the weight matrix in the second fully connected network layer in the first regression network to obtain the second output feature of the target image feature in the second fully connected layer, and the second output feature is activated through the second activation function to obtain the second output feature after activation. The second output feature after activation is weighted through the weight matrix corresponding to the third fully connected layer in the first regression network to obtain the third output feature of the target image feature in the third fully connected layer, and the third activation layer is performed to activate the third output feature to obtain the third output feature after activation. Through the output layer in the first regression network (the output layer can be a fully connected network layer, i.e., the fourth fully connected network layer), the third output features after activation are linearly combined to obtain the brightness enhancement parameters corresponding to the video frame to be processed. In this way, the enhancement parameters on each image information dimension can be obtained through the regression network in the target generation model.
[0148] Step S205 , performing image enhancement on the video frames included in the target video frame set according to the M enhancement parameters, to obtain target image data corresponding to each video frame in the target video frame set.
[0149] Specifically, the computer device can determine the enhancement parameters of the to-be-processed video frame in M image information dimensions as the enhancement parameters corresponding to the target video frame set, and perform image enhancement on the video frames included in the target video frame set according to the M enhancement parameters, to obtain target image data corresponding to each video frame in the target video frame set. In this way, the video frames in the same video frame set use the same set of enhancement parameters, which can not only avoid visible color jumps between video frames in the same scene, but also only need to perform one enhancement parameter prediction for a video frame set, which can greatly reduce the amount of calculation in video applications and improve image processing efficiency.
[0150] Optionally, the target video frame set includes video frame T j; j is a positive integer less than or equal to the number of video frames included in the target video frame set, such as j can be 1, 2, 3, ... The specific method for the computer device to obtain the target image data corresponding to each video frame in the target video frame set may include: j The product of is determined as the video frame T j The first candidate video frame corresponding to the first candidate video frame is obtained. The global average pixel value corresponding to the first candidate video frame is obtained, and the first difference between the pixel value included in the first candidate video frame and the global average pixel value is obtained. The video frame T is determined according to the global average pixel value and the product of the first difference and the contrast enhancement coefficient. j Obtain the channel average pixel value corresponding to the second candidate video frame, obtain the second difference between the pixel value contained in the second candidate video frame and the channel average pixel value, and determine the video frame T according to the channel average pixel value and the product of the second difference and the saturation enhancement coefficient. j The corresponding target image data.
[0151] Specifically, after the computer device obtains the brightness enhancement parameter, contrast enhancement parameter, and saturation enhancement parameter corresponding to the target video frame, the brightness enhancement parameter can be compared with the video frame T j The product of is determined as the first candidate video frame, that is, according to the brightness enhancement parameter, the video frame T j The image brightness information in the image is adjusted to obtain a first candidate video frame. The computer device can also obtain a global average pixel value corresponding to the first candidate video frame, and obtain a first difference between the pixel value contained in the first candidate video frame and the global average pixel value. The global average pixel value is an average pixel value obtained by averaging the pixel values corresponding to all the pixels contained in the first candidate video frame. Therefore, when obtaining the first difference between the pixel value contained in the first candidate video frame and the global average pixel value, the pixel value corresponding to each pixel contained in the first candidate video frame can be calculated, and the difference between the pixel value and the global average pixel value can be calculated, and one pixel corresponds to one first difference. After obtaining the first difference, the computer device can obtain the product of the first difference and the contrast enhancement parameter, and determine the sum of the product of the first difference and the contrast enhancement parameter and the global average pixel value as the video frame T. j The computer device obtains the video frame T jAfter obtaining the corresponding second candidate video frame, the channel average pixel value corresponding to the second candidate video frame can be obtained, and the second difference between the pixel value contained in the second candidate video frame and the channel average pixel value can be obtained. Wherein, each image includes pixel values on three channels, and the channel average pixel value refers to the channel average pixel value on a target channel obtained by averaging the pixels on the three channels. The image corresponding to the second candidate video frame also includes pixel values on three channels. Therefore, obtaining the second difference between the pixel value contained in the second candidate video frame and the channel average pixel value can refer to: calculating the difference between the pixel value on each channel contained in the second candidate video frame and the channel average pixel value on the above target channel, and one pixel point in the second candidate video frame corresponds to a second difference. After obtaining the second difference, the computer device can obtain the product between the second difference and the saturation enhancement parameter, and determine the sum of the product between the second difference and the saturation enhancement parameter and the channel average pixel value as the video frame T. j The corresponding target image data.
[0152] Specifically, the computer device determines the video frame T according to the brightness enhancement parameter. j The corresponding formula of the first candidate video frame can be expressed by formula (1):
[0153] i1=b*I in (1)
[0154] Wherein, I1 in formula (1) refers to the first candidate video frame, b refers to the brightness enhancement parameter, i in refers to the video frame T j .
[0155] Specifically, the computer device determines the video frame T according to the contrast enhancement parameter. j The corresponding formula of the second candidate video frame can be expressed by formula (2):
[0156] I2=c*(I1-mean(I1))+mean(I1) (2)
[0157] Wherein, I2 in formula (2) refers to the input image (i.e., the second candidate video frame), c refers to the contrast enhancement parameter, I1 refers to the first candidate video frame, and mean(I1) refers to the mean function for obtaining the global average pixel value corresponding to the first candidate video frame.
[0158] Specifically, the computer device determines the video frame T according to the saturation enhancement parameter. j The corresponding formula of the target image data can be expressed by formula (3):
[0159] I out=s*(I2-channel_mean(I2))+channel_mean(I2) (3)
[0160] Among them, I in formula (3) out Refers to the output image (i.e., video frame T j Corresponding target image data), s refers to the saturation enhancement parameter, I2 refers to the input image (ie, the second candidate video frame), and channel_mean(I2) refers to a function for obtaining the channel average pixel value corresponding to the second candidate video frame.
[0161] Optionally, it should be noted that before using the above target generation model, it is necessary to train the target generation model. Figures 6 to 8 The training process of the target generation model is described in detail.
[0162] Specifically, the computer device can obtain sample video frames and label information corresponding to the sample video frames, input sample global color features and sample image semantic features corresponding to the sample video frames into the initial generation model, and output the prediction enhancement parameters corresponding to the sample video frames through the initial generation model. According to the label information corresponding to the sample video frames and the prediction enhancement parameters corresponding to the sample video frames, the loss function corresponding to the initial generation model is determined, and according to the loss function, the initial generation model is iteratively trained, and when the initial generation model meets the convergence condition, the initial generation model that meets the convergence condition is determined as the target generation model.
[0163] The computer device can train the initial generation model according to specific needs to obtain a target generation model, such as a target generation model for predicting brightness enhancement parameters, contrast enhancement parameters and saturation enhancement parameters corresponding to video frames. When the computer device has label information corresponding to sample video frames, it can obtain a batch of high-quality color images, determine the high-quality color images as sample video frames, manually perform random color degradation operations on the images and save the coefficients of the random degradation operations, and determine the label information corresponding to the sample video frames based on the coefficients of the random degradation operations. The parameters in the fully connected network layer of the candidate generation model are randomly initialized to obtain the initial generation model, and the sample global color features and sample image semantic features corresponding to the sample video frames are extracted through the fully connected network layer in the initial generation model, and the predicted enhancement parameters corresponding to the sample video frames are output. According to the difference between the label information corresponding to the sample video frames and the predicted enhancement parameters corresponding to the sample video frames, the loss function corresponding to the initial generation model is determined, and according to the loss function, the initial generation model is iteratively trained, and when the initial generation model meets the convergence condition, the initial generation model that meets the convergence condition is determined as the target generation model.
[0164] Among them, the convergence condition corresponding to the initial generation model may refer to that the loss value of the loss function corresponding to the initial generation model is less than the loss threshold, or the loss value is the minimum value of the loss function corresponding to the initial generation model, or the number of training times of the initial generation model reaches the number threshold, etc. For example, when the loss value of the loss function corresponding to the initial generation model is less than the loss threshold, it is determined that the initial generation model meets the convergence condition, and the initial generation model that meets the convergence condition is determined as the target generation model; when the loss value of the loss function corresponding to the initial generation model is greater than or equal to the loss threshold, it is determined that the initial generation model does not meet the convergence condition, and the network parameters in the initial generation model can be adjusted according to the loss value until the initial generation model meets the convergence condition. Specifically, when adjusting the network parameters in the initial generation model according to the loss value, the loss degree to which the loss value belongs can be determined, and the network parameters in the initial generation model can be adjusted according to the loss degree to which the loss value belongs. For example, when it is determined that the loss degree described in the loss value is greater than or equal to the loss threshold, it can be determined that the loss of the initial generation model is large, that is, the difference between the predicted enhancement parameters output by the initial generation model and the manually annotated enhancement parameters is large, and the network parameters in the initial generation model can be adjusted to a greater extent. When it is determined that the loss level described in the loss value is less than the loss threshold, it can be determined that the loss of the initial generation model is small, that is, the difference between the predicted enhancement parameters output by the initial generation model and the manually labeled enhancement parameters is small, and the network parameters in the initial generation model can be adjusted to a small extent. In this way, by adjusting the network parameters in the initial generation model based on the loss value, a greater degree of adjustment can be made when the prediction error of the initial generation model is greater, thereby improving the convergence speed of the initial generation model and the training efficiency. At the same time, it also makes the adjustment operation of the initial generation model more accurate, thereby improving the accuracy of the initial generation model training.
[0165] Among them, the target generation model is used to extract the global color features and semantic features corresponding to the processed video frame, and the enhancement parameters of the processed video frame in the brightness information dimension, contrast information dimension and saturation information dimension are obtained. The network structure of the target generation model can be a fully connected network structure. In the process of training the initial generation, the training and convergence direction of the initial generation model are supervised according to the manually annotated label information, and the training effect is stable and reliable. At the same time, with the manually annotated label information, the control of the adjustment range of the parameters of the initial generation model is highly controllable, so that the prediction effect corresponding to the final target generation model is relatively stable.
[0166] Optionally, during the training process of the above-mentioned initial generation model, a pre-trained feature extraction model can be obtained, through which the semantic features of sample images corresponding to the sample video frames can be output, and the global color features of the samples can be calculated according to the color histogram corresponding to the sample video frames. The semantic features of the sample images and the global color features of the samples are used as input information of the initial generation model. The initial generation model can output predicted enhancement parameters, and the loss function corresponding to the initial generation model is calculated by the difference between the label information corresponding to the sample video frame and the predicted enhancement parameters. The network parameters of the initial generation model are corrected according to the loss function to obtain the trained target generation model. In other words, the initial generation model can be trained on the basis of the trained feature extraction model, which can improve the training efficiency of the initial generation model.
[0167] Optionally, the feature extraction model can also be trained while training the initial generation model. The computer device can obtain the initialized feature extraction model (for the convenience of description, the initialized feature extraction model can be referred to as the initial feature extraction model), and input the acquired sample video frame into the initial feature extraction model. The sample image semantic features corresponding to the sample video frame can be output through the candidate feature extraction model; the sample image semantic features and the sample global color features corresponding to the sample video frame can then be input into the initial generation model, and the prediction enhancement parameters can be output through the initial generation model; based on the sample image semantic features, the prediction enhancement parameters and the label information corresponding to the sample video frame, the total loss function corresponding to the initial feature extraction model and the initial generation model can be calculated, and the network parameters of the initial feature extraction model and the initial generation model are iteratively trained according to the total loss function. When the initial feature extraction model and the initial generation model meet the convergence conditions, the initial feature extraction model that meets the convergence conditions is determined as the feature extraction model, and the initial generation model that meets the convergence conditions is determined as the target generation model. Among them, the specific calculation process of outputting the sample image semantic features corresponding to the sample video frame through the initial feature extraction model is similar to the calculation process of the aforementioned image semantic features, and can be found in Figure 3 The content of step S102 in the embodiment of the present application will not be repeated here. Among them, this solution can perform image enhancement on the video data to be processed through the feature extraction model and the target generation model to obtain the target image data corresponding to the video data to be processed.
[0168] like Figure 6 As shown, Figure 6 is a schematic diagram of a target generation model provided in an embodiment of the present application, such as Figure 6As shown, the computer device can obtain high-quality image data from a batch of high-quality image libraries, and perform color adjustment (such as color degradation or color enhancement) on the high-quality image data, determine the image data after color adjustment as a sample video frame 6a, and use the color adjustment coefficient of the high-quality image data as the label information corresponding to the sample video frame. After obtaining the sample video frame, the computer device can extract image semantic features from the sample video frame 6a through the feature extraction network 6b to obtain the sample image semantic features 6d corresponding to the sample video frame 6a, and obtain the color histogram corresponding to the sample video frame 6a. Figure 6 c. Obtain the sample global color feature 6e corresponding to the sample video frame 6a. Input the sample image semantic feature 6d and the sample global color feature 6e corresponding to the sample video frame 6a into the initial generation model 6f, and generate the prediction enhancement parameter 6g corresponding to the sample video frame 6a through the initial generation model 6f. According to the prediction enhancement parameter output by the initial generation model 6f and the label information corresponding to the sample video frame 6a, determine the loss function 6h corresponding to the initial generation model 6f, adjust the parameters of the initial generation model 6f according to the loss function 6h, and when the initial generation model 6f meets the convergence condition, determine the target generation model 6i with the initial generation model 6f that meets the convergence condition. Among them, the convergence condition corresponding to the initial generation model 6f may refer to the loss value of the loss function corresponding to the initial generation model 6f being less than the loss threshold, or the loss value being the minimum value of the loss function corresponding to the initial generation model 6f, or the number of training times of the initial generation model 6f reaching the number threshold, etc. For example, when the loss value of the loss function corresponding to the initial generation model 6f is less than the loss threshold, it is determined that the initial generation model 6f meets the convergence condition, and the initial generation model 6f that meets the convergence condition is determined as the target generation model; when the loss value of the loss function corresponding to the initial generation model 6f is greater than or equal to the loss threshold, the network parameters in the initial generation model 6f are adjusted according to the loss value until the initial generation model 6f meets the convergence condition.
[0169] like Figure 7 As shown, Figure 7 is a schematic diagram of prediction enhancement parameters corresponding to a target generation model provided in an embodiment of the present application, such as Figure 7 As shown, the target generation model is used to predict the enhancement parameters of the video frame to be processed, and the comparison diagrams of the brightness prediction enhancement parameters, contrast prediction enhancement parameters and saturation prediction enhancement parameters corresponding to the video frame to be processed and the marked values are obtained. Figure 7As shown, in the comparison diagram of the brightness prediction enhancement parameter and the brightness annotation value, it can be seen that the brightness prediction enhancement parameters corresponding to each video frame in the video data to be processed obtained by the target generation model are all around the annotation value, and the mean of the brightness prediction enhancement parameters corresponding to each video frame is very close to the mean of the annotation value, and the variance is also within the acceptable threshold range. Similarly, in the comparison diagram of the contrast prediction enhancement parameter and the contrast annotation value, the contrast prediction enhancement parameters corresponding to each video frame in the video data to be processed obtained by the target generation model are all around the annotation value, and the mean of the contrast prediction enhancement parameters corresponding to each video frame is very close to the mean of the annotation value, and the variance is also within the acceptable threshold range. Similarly, in the comparison diagram of the saturation prediction enhancement parameter and the saturation annotation value, the saturation prediction enhancement parameters corresponding to each video frame in the video data to be processed obtained by the target generation model are all around the annotation value, and the mean of the saturation prediction enhancement parameters corresponding to each video frame is very close to the mean of the annotation value, and the variance is also within the acceptable threshold range.
[0170] like Figure 8 As shown, Figure 8 is a schematic diagram of a comparison between a video frame to be processed and corresponding target image data provided by an embodiment of the present application, such as Figure 8 As shown, image 8a refers to the reference image, and the corresponding manual annotation value of the reference image is 0.76, that is, the target image is to be obtained through the target generation model, image 8b refers to the image to be processed, that is, the original image to be enhanced, and image 8c refers to the target image, that is, the target image obtained after the image to be processed is enhanced, and the corresponding prediction value of image 8c is 0.9. Figure 8 As shown, the original image 8c achieves a better image enhancement effect than the reference image 8a. This is because the predicted value of the image enhancement parameter is a relatively accurate threshold obtained by fitting after referring to a large number of different types of high-quality color maps. Therefore, the effect of the target image after image enhancement may be better than that of the reference image. Therefore, by performing image enhancement on the image to be processed in the embodiment of the present application, the display effect of the image to be processed can be improved.
[0171] After the initial generation model is trained as described above and a trained target generation model is obtained, the target generation model can be applied to live video enhancement application scenarios, on-demand video enhancement application scenarios, image optimization application scenarios and other scenarios.
[0172] like Fig. 9 As shown, Fig. 9 is a schematic diagram of a video image processing scenario provided by an embodiment of the present application, such as Fig. 9As shown, after user A shoots a user video 9a, if he wants to upload the user video data 9a to a short video sharing platform and share his life video with others, he can upload the user video data 9a to the user terminal A1 for image enhancement provided by the present application, perform image enhancement on the user video data 9a shot by user A, and improve the video quality corresponding to the user video data 9a, so as to reduce problems such as dark images or image exposure in the user video 9a, and improve the viewing experience of the viewing user. After the user terminal A1 obtains the user video data 9a, it can divide the user video data 9a into scenes to obtain N video frame sets 9b, where N is a positive integer. The N video frame sets 9b include video frame set 1, video frame set 2...video frame set N, and each video frame set includes one or more video frames. For the specific division method, please refer to Figure 3 The content of step S101 in this embodiment of the present application will not be repeated here.
[0173] Furthermore, when the user terminal A1 divides the video data to be processed and obtains N video frame sets, the N video frame sets can be labeled with identifiers with a sequence, such as 1, 2, 3... or a, b, c..., etc., to label the N video frame sets, and obtain video frame set 1, video frame set 2... video frame set N. The user terminal A1 can perform image processing on the N video frame sets in sequence according to the sequence of the identifiers of each video frame set. For example, taking the target video frame set 9c as an example, the target video frame set 9c can be any one of the N video frame sets, and a video frame is randomly selected from the video frames included in the target video frame set 9c as the video frame to be processed 9d. The video frame to be processed 9d is input into the semantic extraction network 9e, and image semantic features are extracted for the video frame to be processed 9d to obtain image semantic features 9g corresponding to the video frame to be processed 9d. The video frame to be processed 9d is input into the color feature extraction network 9f to obtain the color histogram corresponding to the video frame to be processed 9d. According to the color histogram corresponding to the video frame to be processed 9d, the global color feature 9h corresponding to the video frame to be processed 9d is obtained. For details on how to obtain the color histogram corresponding to the video frame to be processed 9d and obtain the image semantic feature 9g corresponding to the video frame to be processed 9d, please refer to Figure 3 The content of step S101 in the embodiment of the present application will not be repeated here. After the user terminal A1 obtains the global color feature 9h and the image semantic feature 9g corresponding to the video frame 9d to be processed, the global color feature 9h and the image semantic feature 9g can be input into the target generation model 9i, and the enhancement parameter 9j corresponding to the video frame 9d to be processed can be generated by the target generation model 9i. The structure of the target generation model can be found in Figure 4 The content of step S204 in this embodiment of the present application will not be repeated here.
[0174] After the user terminal A1 obtains the enhancement parameter corresponding to the video frame 9d to be processed, the enhancement parameter corresponding to the video frame 9d to be processed can be used to determine the enhancement parameter corresponding to the target video frame set 9c, and the video frames contained in the target video frame set 9c are image enhanced to obtain the target image data 9k corresponding to each video frame in the target video frame set 9c. Such a video frame set only obtains the enhancement parameter once, that is, the enhancement parameter corresponding to the video frame in a video frame set is determined as the enhancement parameter of the corresponding video frame set, which can improve the calculation efficiency and the image enhancement efficiency. In this way, the above-mentioned method of image enhancement of the video frame in the target video frame set is adopted to obtain the target image data corresponding to the video frames in the N video frame sets 9b, and then obtain the target image data corresponding to the user video data 9a. When the user terminal 1 performs image enhancement on the user video data 9a corresponding to the user A and obtains the target image data corresponding to the user video data 9a, the user can store the target image data corresponding to the user video data 9a, and can upload the target image data corresponding to the user video data 9a to the short video platform to be uploaded. It can be seen that the embodiments of the present application can perform image enhancement on user video data, and can solve problems such as poor video quality (such as overexposure or excessive darkness) caused by improper user operation. The video quality of user video data can be improved to improve the viewing experience of the viewing user, and the image enhancement efficiency is high.
[0175] In the embodiment of the present application, a target video frame set can be obtained from the video data to be processed, wherein the video frames included in the target video frame set have the same attribute information, the video frames to be processed are obtained from the target video frame set, the global color features corresponding to the video frames to be processed are obtained, and the image semantic features corresponding to the video frames to be processed are obtained. According to the global color features and the image semantic features, the enhancement parameters of the video frames to be processed in M image information dimensions are obtained, and according to the M enhancement parameters, the video frames included in the target video frame set are image enhanced to obtain the target image data corresponding to each video frame in the target video frame set. In the present application, the video data to be processed is divided according to the color histogram corresponding to each video frame to obtain N video frame sets, wherein the video frames included in each video frame set have the same attribute information, so that the consistency of the division of the video data to be processed can be guaranteed, that is, the video frames included in each video frame set are guaranteed to have the same attribute information (such as the video frames in the same video frame set all belong to the same scene screen). In this way, when acquiring enhancement parameters for each video frame set, a video frame can be directly randomly acquired from each video frame set as the video frame to be processed, and the enhancement coefficients of the video frame to be processed in M image information dimensions are acquired according to the global color features and image semantic features corresponding to the video frame to be processed, and the enhancement coefficients are used as the enhancement coefficients corresponding to the corresponding video frame set, and the video frames included in the video frame set are respectively enhanced to obtain the target image data corresponding to each video frame in each video frame set. It can be seen that it is only necessary to acquire enhancement parameters for one video frame in a video frame set, so as to determine the enhancement parameters corresponding to the video frame as the enhancement parameters of the corresponding video frame set, which can reduce the amount of calculation and improve the efficiency of image enhancement. At the same time, video frames with the same attributes can use the same set of enhancement parameters, which can reduce the risk of color jump phenomenon in the image, and thus improve the enhancement quality of the video data to be processed. In addition, a fully connected regression network is used in the embodiment of the present application. According to the global color features and image semantic features corresponding to the video frame to be processed, enhancement parameters corresponding to the video frame to be processed are obtained, and image enhancement is performed on each video frame in the target video frame set according to the enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set. The image enhancement effect is stable and controllable, and the image enhancement efficiency is high.
[0176] See also Fig.10 , Fig.10 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.10 As shown, the above-mentioned image processing device can be applied to the above-mentioned Figure 1The client on any user terminal in the user terminal cluster shown may be a computer program (including program code) in a computer device. The image processing device may include: a first acquisition module 11, a second acquisition module 12, a third acquisition module 13, and an image enhancement module 14;
[0177] The first acquisition module 11 is used to acquire a target video frame set from the video data to be processed; the video frames included in the target video frame set have the same attribute information;
[0178] The second acquisition module 12 is used to acquire a to-be-processed video frame from the target video frame set, acquire a global color feature corresponding to the to-be-processed video frame, and acquire an image semantic feature corresponding to the to-be-processed video frame;
[0179] The third acquisition module 13 is used to acquire enhancement parameters of the to-be-processed video frame in M image information dimensions respectively according to the global color features and the image semantic features; M is a positive integer;
[0180] The image enhancement module 14 is used to perform image enhancement on the video frames included in the target video frame set according to M enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set.
[0181] Wherein, the first acquisition module 11 includes:
[0182] The first acquisition unit 1101 is used to acquire the video data to be processed, divide the video data to be processed into L video frames, and acquire the color histograms corresponding to the L video frames respectively; the L video frames include video frames T having adjacent relationships. i and video frame T i+1 , L is a positive integer;
[0183] The second acquisition unit 1102 is used to obtain the video frame T i and video frame T i+1 The corresponding color histograms are obtained to obtain the video frame T i and video frame T i+1 The similarity distance between them; i is a positive integer less than L;
[0184] The division unit 1103 is used to divide the L video frames according to the similarity distance to obtain N video frame sets; the similarity distance between adjacent video frames in each video frame set is less than a distance threshold, and N is a positive integer;
[0185] The selection unit 1104 is configured to select any one video frame set from the N video frame sets as a target video frame set.
[0186] The first acquisition unit 1101 is specifically used for:
[0187] Divide the color space into P image color intervals and obtain video frame T in L video frames i ; P is a positive integer;
[0188] According to the video frame T i The image color space where the pixels are located in is counted, and the number of pixels contained in each image color interval is counted;
[0189] Generate video frame T according to the number of pixels corresponding to each of the P image color intervals i The corresponding color histogram.
[0190] Wherein, the second acquisition module 12 includes:
[0191] The third acquisition unit 1201 is used to resize the video frame to be processed to obtain a candidate video frame with a target size, and obtain a global color feature corresponding to the video frame to be processed according to a color histogram corresponding to the candidate video frame;
[0192] The convolution operation unit 1202 is used to input the candidate video frame into the feature extraction model, and perform a convolution operation on the candidate video frame according to the convolution layer in the feature extraction model to obtain the image convolution feature corresponding to the candidate video frame;
[0193] The residual operation unit 1203 is used to perform residual operation on the image convolution feature according to the residual layer in the feature extraction model to obtain the image semantic feature corresponding to the video frame to be processed.
[0194] Wherein, the third acquisition module 13 includes:
[0195] A splicing processing unit 1301 is used to splice the global color feature and the image semantic feature to obtain the target image feature;
[0196] The first output unit 1302 is used to input the target image features into the target generation model, and output enhancement parameters of the target image features in M image information dimensions respectively through the target generation model.
[0197] Among them, the enhancement parameters on the M image information dimensions include brightness enhancement parameters, contrast enhancement parameters and saturation enhancement parameters;
[0198] The first output unit 1302 is specifically used for:
[0199] Inputting the target image features into the target generation model, performing weighted processing on the target image features according to the weight matrix corresponding to the first regression network in the target generation model, and obtaining the brightness enhancement parameters corresponding to the video frame to be processed;
[0200] According to the weight matrix corresponding to the second regression network in the target generation model, the target image features are weighted to obtain the contrast enhancement parameters corresponding to the video frame to be processed;
[0201] According to the weight matrix corresponding to the third regression network in the target generation model, the target image features are weighted to obtain the saturation enhancement parameters corresponding to the video frame to be processed.
[0202] The target video frame set includes video frames T j ; j is a positive integer less than or equal to the number of video frames included in the target video frame set;
[0203] The image enhancement module 14 includes:
[0204] The first determining unit 1401 is used to determine the brightness enhancement parameter and the video frame T j The product of is determined as the video frame T j a corresponding first candidate video frame;
[0205] The second determining unit 1402 is used to obtain the global average pixel value corresponding to the first candidate video frame, obtain the first difference between the pixel value included in the first candidate video frame and the global average pixel value, and determine the video frame T according to the global average pixel value and the product of the first difference and the contrast enhancement coefficient. j a corresponding second candidate video frame;
[0206] The third determining unit 1403 is used to obtain the channel average pixel value corresponding to the second candidate video frame, obtain the second difference between the pixel value included in the second candidate video frame and the channel average pixel value, and determine the video frame T according to the channel average pixel value and the product of the second difference and the saturation enhancement coefficient. j The corresponding target image data.
[0207] The third acquisition module 13 further includes:
[0208] The fourth acquisition unit 1303 is used to acquire sample video frames and label information corresponding to the sample video frames;
[0209] The second output unit 1304 is used to input the sample global color features and the sample image semantic features corresponding to the sample video frame into the initial generation model, and output the prediction enhancement parameters corresponding to the sample video frame through the initial generation model;
[0210] The fourth determining unit 1305 is used to determine the loss function corresponding to the initial generation model according to the label information corresponding to the sample video frame and the prediction enhancement parameter corresponding to the sample video frame;
[0211] The fifth determining unit 1306 is used to iteratively train the initial generation model according to the loss function, and when the initial generation model meets the convergence condition, determine the initial generation model that meets the convergence condition as the target generation model.
[0212] According to one embodiment of the present application, Figure 3 The steps involved in the image processing method shown can be represented by Fig.10 The various modules in the image processing device shown are executed. For example, Figure 3 The step S101 shown in FIG. 1 can be performed by Fig.10 The first acquisition module 11 is executed, Figure 3 The step S102 shown in FIG. 1 can be performed by Fig.10 The second acquisition module 12 is executed, Figure 3 The step S103 shown in FIG. 1 can be performed by Fig.10 The third acquisition module 13 is executed, Figure 3 The step S104 shown in FIG. 1 can be performed by Fig.10 The image enhancement module 14 in is used to perform the above.
[0213] According to one embodiment of the present application, Fig.10 The various modules in the image processing device shown can be respectively or all merged into one or several units to constitute, or some of the units (some) can also be split into multiple smaller sub-units in function, and the same operation can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the function of a module can also be realized by multiple units, or the function of multiple modules can be realized by one unit. In other embodiments of the present application, the test device can also include other units. In practical applications, these functions can also be assisted by other units and can be realized by the collaboration of multiple units.
[0214] In the embodiment of the present application, a target video frame set can be obtained from the video data to be processed, wherein the video frames included in the target video frame set have the same attribute information, the video frames to be processed are obtained from the target video frame set, the global color features corresponding to the video frames to be processed are obtained, and the image semantic features corresponding to the video frames to be processed are obtained. According to the global color features and the image semantic features, the enhancement parameters of the video frames to be processed in M image information dimensions are obtained, and according to the M enhancement parameters, the video frames included in the target video frame set are image enhanced to obtain the target image data corresponding to each video frame in the target video frame set. In the present application, the video data to be processed is divided according to the color histogram corresponding to each video frame to obtain N video frame sets, wherein the video frames included in each video frame set have the same attribute information, so that the consistency of the division of the video data to be processed can be guaranteed, that is, the video frames included in each video frame set are guaranteed to have the same attribute information (such as the video frames in the same video frame set all belong to the same scene screen). In this way, when acquiring enhancement parameters for each video frame set, a video frame can be directly randomly acquired from each video frame set as the video frame to be processed, and the enhancement coefficients of the video frame to be processed in M image information dimensions are acquired according to the global color features and image semantic features corresponding to the video frame to be processed, and the enhancement coefficients are used as the enhancement coefficients corresponding to the corresponding video frame set, and the video frames included in the video frame set are respectively enhanced to obtain the target image data corresponding to each video frame in each video frame set. It can be seen that it is only necessary to acquire enhancement parameters for one video frame in a video frame set, so as to determine the enhancement parameters corresponding to the video frame as the enhancement parameters of the corresponding video frame set, which can reduce the amount of calculation and improve the efficiency of image enhancement. At the same time, video frames with the same attributes can use the same set of enhancement parameters, which can reduce the risk of color jump phenomenon in the image, and thus improve the enhancement quality of the video data to be processed. In addition, a fully connected regression network is used in the embodiment of the present application. According to the global color features and image semantic features corresponding to the video frame to be processed, enhancement parameters corresponding to the video frame to be processed are obtained, and image enhancement is performed on each video frame in the target video frame set according to the enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set. The image enhancement effect is stable and controllable, and the image enhancement efficiency is high.
[0215] See also Fig.11 , Fig.11 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.11As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001. As Fig.11 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.
[0216] In such Fig.11 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:
[0217] A target video frame set is obtained from the video data to be processed; the video frames included in the target video frame set have the same attribute information;
[0218] Obtain a video frame to be processed from the target video frame set, obtain a global color feature corresponding to the video frame to be processed, and obtain an image semantic feature corresponding to the video frame to be processed;
[0219] According to the global color features and image semantic features, the enhancement parameters of the video frame to be processed in M image information dimensions are obtained; M is a positive integer;
[0220] According to the M enhancement parameters, image enhancement is performed on the video frames included in the target video frame set to obtain target image data corresponding to each video frame in the target video frame set.
[0221] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 The description of the image processing method in the corresponding embodiment can also be performed as described above. Fig.10 The description of the image processing device in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.
[0222] In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, program instructions can be deployed on a computing device for execution, or executed on multiple computing devices located in one location, or executed on multiple computing devices distributed in multiple locations and interconnected by a communication network. Multiple computing devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.
[0223] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device executes the above Figure 3 or Figure 4 Therefore, the description of the image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.
[0224] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.
[0225] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.
[0226] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.
[0227] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0228] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. An image processing method, characterized in that: include: Obtaining video data to be processed, and obtaining a target video frame set from the video data to be processed according to similarity distances between adjacent video frames in the video data to be processed; the video frames included in the target video frame set have the same attribute information; Obtaining a to-be-processed video frame from the target video frame set, and obtaining global color features and image semantic features corresponding to the to-be-processed video frame; The global color feature and the image semantic feature are spliced into a target image feature, the target image feature is input into a target generation model, and the parameter of the target image feature is predicted by M regression networks in the target generation model to obtain enhancement parameters of the to-be-processed video frame in M image information dimensions, wherein one image information dimension corresponds to one regression network, and M is a positive integer; According to the M enhancement parameters, image enhancement is performed on the video frames included in the target video frame set to obtain target image data corresponding to each video frame in the target video frame set.
2. The method according to claim 1, characterized in that The step of obtaining a target video frame set from the video data to be processed according to similarity distances between adjacent video frames in the video data to be processed includes: The video data to be processed is divided into L video frames, and the color histograms corresponding to the L video frames are obtained; the L video frames include video frames T having an adjacent relationship. i and video frame T i+1 , L is a positive integer; According to the video frame T i and the video frame T i+1 The corresponding color histograms are obtained to obtain the video frame T i and the video frame T i+1 The similarity distance between them; i is a positive integer less than L; If the video frame T i and the video frame T i+1 If the similarity distance between them is greater than or equal to the distance threshold, the video frame T i and the video frame T i+1 Divide into different sets of video frames; If the video frame T i and the video frame T i+1 If the similarity distance between them is less than the distance threshold, the video frame T i and the video frame T i+1 Divide into the same set of video frames; When the L video frames are all divided into corresponding video frame sets, N video frame sets are obtained, and any one video frame set is selected from the N video frame sets as the target video frame set; N is a positive integer.
3. The method according to claim 1, characterized in that: The obtaining of global color features and image semantic features corresponding to the video frame to be processed includes: Resizing the video frame to be processed to obtain a candidate video frame with a target size; Obtaining a color histogram corresponding to the candidate video frame, performing color feature extraction on the color histogram corresponding to the candidate video frame, and obtaining a global color feature corresponding to the video frame to be processed; Inputting the candidate video frame into a feature extraction model, and performing a convolution operation on the candidate video frame according to a convolution layer in the feature extraction model to obtain an image convolution feature corresponding to the candidate video frame; According to the residual layer in the feature extraction model, a residual operation is performed on the image convolution feature to obtain the image semantic feature corresponding to the video frame to be processed.
4. The method according to claim 1, characterized in that The M image information dimensions include a brightness information dimension, a contrast information dimension, and a saturation information dimension; the M regression networks in the target generation model include a first regression network corresponding to the brightness information dimension, a second regression network corresponding to the contrast information dimension, and a third regression network corresponding to the saturation information dimension; The method of performing parameter prediction on the target image features through the M regression networks in the target generation model to obtain enhancement parameters of the to-be-processed video frame in M image information dimensions respectively includes: According to the weight matrix corresponding to the first regression network, weighted processing is performed on the target image features to obtain a brightness enhancement parameter corresponding to the video frame to be processed; the brightness enhancement parameter is used to characterize the enhancement parameter of the video frame to be processed in the brightness information dimension; According to the weight matrix corresponding to the second regression network, weighted processing is performed on the target image features to obtain contrast enhancement parameters corresponding to the video frame to be processed; the contrast enhancement parameters are used to characterize enhancement parameters of the video frame to be processed in the contrast information dimension; According to the weight matrix corresponding to the third regression network, the target image features are weightedly processed to obtain the saturation enhancement parameters corresponding to the video frame to be processed; the saturation enhancement parameters are used to characterize the enhancement parameters of the video frame to be processed in the saturation information dimension.
5. The method according to claim 4, characterized in that The step of performing weighted processing on the target image features according to the weight matrix corresponding to the first regression network to obtain the brightness enhancement parameter corresponding to the video frame to be processed includes: Performing weighted processing on the target image feature through a weight matrix corresponding to a first fully connected layer in the first regression network to obtain a first output feature, and performing activation processing on the first output feature through a first activation layer in the first regression network to obtain a first output feature after activation processing; Performing weighted processing on the first output feature after activation through the weight matrix corresponding to the second fully connected layer in the first regression network to obtain a second output feature, and performing activation processing on the second output feature through the second activation layer in the first regression network to obtain the second output feature after activation; Performing weighted processing on the second output feature after the activation processing through the weight matrix corresponding to the third fully connected layer in the first regression network to obtain a third output feature, and performing activation processing on the third output feature through the third activation layer in the first regression network to obtain the third output feature after the activation processing; Through the output layer in the first regression network, the third output features after the activation process are linearly combined to obtain the brightness enhancement parameters corresponding to the video frame to be processed.
6. The method according to claim 4, characterized in that The target video frame set includes video frames T j ; j is a positive integer less than or equal to the number of video frames included in the target video frame set; The performing image enhancement on the video frames included in the target video frame set according to the M enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set includes: The brightness enhancement parameter is combined with the video frame T j The product of is determined as the video frame T j a corresponding first candidate video frame; Obtain a global average pixel value corresponding to the first candidate video frame, obtain a first difference between a pixel value included in the first candidate video frame and the global average pixel value, and determine the video frame T according to the global average pixel value and the product of the first difference and the contrast enhancement coefficient. j a corresponding second candidate video frame; Obtain a channel average pixel value corresponding to the second candidate video frame, obtain a second difference between the pixel value contained in the second candidate video frame and the channel average pixel value, and determine the video frame T according to the channel average pixel value and the product of the second difference and the saturation enhancement coefficient. j The corresponding target image data.
7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining sample video frames and label information corresponding to the sample video frames; Obtaining a sample global color feature and a sample image semantic feature corresponding to the sample video frame, and inputting the sample global color feature and the sample image semantic feature into an initial generation model; Through the M regression networks in the initial generation model, parameter prediction is performed on the sample global color features and the sample image semantic features to obtain prediction enhancement parameters of the sample video frame in M image information dimensions respectively; M is a positive integer; Determining a loss function corresponding to the initial generation model according to a difference between the label information and the predicted enhancement parameter; According to the loss function, the initial generation model is iteratively trained, and when the initial generation model meets the convergence condition, the initial generation model that meets the convergence condition is determined as the target generation model.
8. The method according to claim 7, characterized in that The tag information includes coefficients of a random color degradation operation performed on the sample video frame.
9. An image processing device, characterized in that: include: A first acquisition module is used to acquire the video data to be processed, and acquire a target video frame set from the video data to be processed according to the similarity distance between adjacent video frames in the video data to be processed; the video frames included in the target video frame set have the same attribute information; A second acquisition module is used to acquire a to-be-processed video frame from the target video frame set, and acquire a global color feature and an image semantic feature corresponding to the to-be-processed video frame; A third acquisition module is used to splice the global color feature and the image semantic feature into a target image feature, input the target image feature into a target generation model, and perform parameter prediction on the target image feature through M regression networks in the target generation model to obtain enhancement parameters of the to-be-processed video frame in M image information dimensions, wherein one image information dimension corresponds to one regression network, and M is a positive integer; The image enhancement module is used to perform image enhancement on the video frames included in the target video frame set according to M enhancement parameters to obtain target image data corresponding to each video frame in the target video frame set.
10. A computer device, characterized in that: including memory and processor; The memory is connected to the processor, the memory is used to store the computer program, and the processor is used to call the computer program so that the computer device executes the method described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: A computer program is stored in the computer-readable storage medium, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having a processor executes the method according to any one of claims 1 to 8.
12. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 8.
