Image processing method, device, equipment and computer readable storage medium
By obtaining the picture layout information of the image to be processed and selecting the corresponding image segmentation model, the problem of size and direction mismatch in image preprocessing is solved, the accuracy of image segmentation and processing is improved, and the accuracy and quality of image processing are ensured.
Patent Information
- Application Number
- CN202010744832.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-29
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-07-29
AI Technical Summary
In the prior art, when generating portrait masks, due to the mismatch of size and direction in image preprocessing, the accuracy of the cutout mask output by the neural network is reduced, resulting in a decrease in image processing accuracy.
By obtaining the picture layout information of the image to be processed, determining its target picture direction type, and selecting the corresponding image segmentation model for image segmentation, ensuring that the default layout direction of the image segmentation model matches the image, reducing distortion, and thus improving the image segmentation result Accuracy.
The accuracy of the image segmentation model is improved, and the accuracy and quality of image processing based on the segmentation result are improved.
Smart Images

Figure CN111862124B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, apparatus, device and computer-readable storage medium. Background Art
[0002] At present, for the generation method of the mask for blurring and cutting out the portrait, the image segmentation or portrait background separation method is usually used to cut out the portrait, and the result is used for blur post-processing to produce an effect similar to that of SLR photography. When the portrait mask is generated by a neural network according to the above-mentioned image processing algorithm, the image to be processed is usually preprocessed first, and its screen layout is adjusted to the default layout direction of the neural network before being input into the neural network. When the layout direction of the image to be processed is inconsistent with the default layout direction of the neural network, the preprocessed image will be distorted, resulting in a reduction in the accuracy of the portrait mask output by the final neural network for image segmentation of the preprocessed image, thereby reducing the accuracy of further image processing based on the portrait mask. Summary of the invention
[0003] The embodiments of the present application provide an image processing method, an apparatus, and a computer-readable storage medium, which can improve the image processing accuracy.
[0004] The technical solution of the embodiment of the present application is implemented as follows:
[0005] The present application provides an image processing method, including:
[0006] Acquire an image to be processed and screen layout information of the image to be processed; the screen layout information includes a picture shooting direction and an image size;
[0007] Determining a target screen direction type of the image to be processed according to the screen layout information of the image to be processed;
[0008] According to the correspondence between the preset picture direction type and the image segmentation model, determining a target image segmentation model corresponding to the target picture direction type from at least two preset image segmentation models;
[0009] Using the target image segmentation model to perform image segmentation on the image to be processed to obtain an image segmentation result;
[0010] Image processing is performed based on the image segmentation result to obtain a final processed image.
[0011] The present application provides an image processing device, including:
[0012] An acquisition unit, used to acquire an image to be processed and screen layout information of the image to be processed; the screen layout information includes a picture shooting direction and / or an image size;
[0013] A determination unit, configured to determine a target screen direction type of the image to be processed according to screen layout information of the image to be processed;
[0014] A selection unit, configured to determine a target image segmentation model corresponding to the target picture direction type from at least two preset image segmentation models according to a correspondence between preset picture direction types and image segmentation models;
[0015] A segmentation unit, used to perform image segmentation on the preprocessed image using the target image segmentation model to obtain an image segmentation result;
[0016] A processing unit is used to perform image processing based on the image segmentation result to obtain a final processed image.
[0017] The present application provides an image processing device, characterized in that it includes:
[0018] Memory for storing computer programs;
[0019] The processor is used to implement the image processing method provided in the embodiment of the present application when executing the computer program stored in the memory.
[0020] An embodiment of the present application provides a computer-readable storage medium storing a computer program for causing a processor to execute the image processing method provided in the embodiment of the present application.
[0021] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:
[0022] An image processing method, device, and computer-readable storage medium provided by the embodiments of the present application can select a corresponding target image segmentation model to perform image segmentation operations according to the target screen direction type of the image to be processed. For example, when the target screen direction type is a horizontal type, a horizontal type image segmentation model can be selected to perform image segmentation on the image to be processed; when the target screen direction type is a vertical type, a vertical type image segmentation model can be selected to perform image segmentation on the image to be processed. In this way, it can be ensured that the target screen direction type of the image to be processed matches the default layout direction of the corresponding image segmentation model, reducing preprocessing operations that may cause distortion, thereby ensuring the accuracy of the image segmentation results obtained by the image segmentation model, and thereby improving the accuracy of image processing based on the image segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is an optional structural diagram of the image processing system architecture provided in the embodiment of the present application;
[0024] Figure 2 is an optional structural diagram of an image processing device provided in an embodiment of the present application;
[0025] Figure 3 It is an optional flowchart of the image processing method provided in the embodiment of the present application;
[0026] Figure 4 It is an optional flowchart of the image processing method provided in the embodiment of the present application;
[0027] Figure 5 It is an optional flowchart of the image processing method provided in the embodiment of the present application;
[0028] FIG. 6( a ) is a schematic diagram of a picture shooting direction provided in an embodiment of the present application Figure 1 ;
[0029] FIG. 6( b ) is a schematic diagram of a picture shooting direction provided in an embodiment of the present application Figure 2 ;
[0030] FIG. 6( c ) is a schematic diagram of a picture shooting direction provided in an embodiment of the present application Figure 3 ;
[0031] FIG. 6( d ) is a schematic diagram of a picture shooting direction provided in an embodiment of the present application Figure 4 ;
[0032] Figure 7 It is an optional flowchart of the image processing method provided in the embodiment of the present application;
[0033] Figure 8 is a schematic diagram of an image segmentation result provided in an embodiment of the present application;
[0034] Fig. 9 It is an optional flowchart of the image processing method provided in the embodiment of the present application;
[0035] Fig.10 It is an optional flowchart of the image processing method provided in the embodiment of the present application;
[0036] Fig.11 is a schematic diagram of a final processed image provided by an embodiment of the present application;
[0037] FIG. 12( a ) is a schematic diagram of an optional flow chart of an image processing method provided in an embodiment of the present application;
[0038] FIG12( b ) is a schematic diagram of an optional flow chart of an image processing method provided in an embodiment of the present application;
[0039] Fig.13It is an optional flowchart of the image processing method provided in the embodiment of the present application. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0041] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0042] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.
[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0044] At present, for the application scenario of generating a cutout mask, for example, a portrait mask is generated by cutout of a portrait from an original image. Usually, a portrait segmentation or a portrait background separation (matting) method is used to cut out the portrait, and a portrait mask is obtained. The portrait mask is used for subsequent image processing, and an image with a blurred background is generated according to the portrait mask, thereby producing an effect similar to that of a SLR camera.
[0045] At present, for generating a cutout mask such as a portrait mask, a convolutional neural network (CNN) is generally used to process the input image and output a cutout mask corresponding to the input image. Before the original image is input into the CNN network model, the original photographed image needs to be preprocessed, including scaling, padding, normalization, rotation and other processes. Among them, the scaling process is to reduce the photographed image to the size required for the CNN network input, generate a mask through the CNN network inference, and then enlarge the mask to the original image size. The filling process is when the aspect ratio of the photographed image size is inconsistent with the aspect ratio of the CNN network input size, by filling outside the edge of the photographed image to make the two ratios consistent. The normalization process is to normalize the three RGB channels of the image by subtracting the mean and dividing the variance, so that the CNN network converges effectively and prevents the network from overfitting. The rotation process is for images of different shooting directions generated by different mobile phone directions when taking pictures, and the image needs to be rotated to the shooting direction preset by the CNN network.
[0046] It can be seen that the preprocessing operation method currently used for the original image in order to generate a cutout mask using the CNN network requires that the image be reduced to a fixed size ratio before being input into the network. For example, the CNN network requires the input image size ratio to be 1:1, the original horizontal image aspect ratio is 4:3, and the original vertical image aspect ratio is 3:4. Then, after the original horizontal image and the original vertical image are reduced to a ratio of 1:1, the original horizontal and vertical images will both be distorted due to the inconsistent ratios, resulting in reduced accuracy of the cutout mask output by the CNN network, and edge misalignment, holes, false detection, etc. may occur. Similarly, if the CNN network requires the input image size ratio to be 3:4, the original horizontal image aspect ratio is 4:3, and the original vertical image aspect ratio is 3:4, then after reduction, only the original vertical image can be guaranteed to be undistorted, and the horizontal image will be distorted, and vice versa. The vertical image will be distorted. In addition, although the current method of filling outside the edge of the original image can make the filled image consistent with the CNN network input ratio, it will cause the proportion of the effective part of the image to be reduced, and will still cause the accuracy of the output cutout mask to be reduced.
[0047] The embodiments of the present application provide an image processing method, apparatus, device, and computer-readable storage medium, which can improve the image processing accuracy. The following describes an exemplary application of the image processing device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as various types of user terminals such as smart phones, tablet computers, and laptop computers, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a terminal.
[0048] See also Figure 1 , Figure 1This is an optional architecture diagram of the image processing system 100 provided in an embodiment of the present application. To support an image processing application, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The image processing system is described below by taking the scenario in which the first user transmits a picture taken and processed with background blur to the second user in a social application as an example. The first user corresponds to the terminal 400-1, and the second user corresponds to the terminal 400-2. When implementing the scenario of image processing and image transmission, terminal 400-1 is used to obtain the image to be processed, and determine the target screen direction type of the image to be processed according to the image size of the image to be processed; according to the correspondence between the preset screen direction type and the image segmentation model, determine the target image segmentation model corresponding to the target screen direction type from at least two preset image segmentation models; the target image segmentation model is used to segment the target image from the image to be processed; at least two image segmentation models correspond one-to-one to at least two screen direction types; the target image segmentation model is used to perform image segmentation on the pre-processed image to obtain an image segmentation result; image processing is performed based on the image segmentation result to obtain a final processed image, such as an image with a blurred background effect; and the final processed image is displayed on the graphical interface 401-1.
[0049] The server 200 is used to send the final processed image to the terminal 400 - 2 when the first user sends the final processed image through the terminal 400 - 1 , so that the terminal 400 - 2 displays the final processed image on the graphical interface 401 - 2 ; the database 500 is used to store the final processed image in the server 200 .
[0050] See also Figure 2 , Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application, Figure 2 The terminal 400 shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .
[0051] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0052] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0053] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 450 optionally includes one or more storage devices that are physically far away from the processor 410.
[0054] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0055] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0056] A network communication module 452, for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);
[0057] a display module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screens, speakers, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);
[0058] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.
[0059] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 An image processing device 455 stored in a memory 450 is shown, which may be software in the form of a program and a plug-in, etc., and includes the following software modules: an acquisition unit 4551, a determination unit 4552, a selection unit 4553, a segmentation unit 4554 and a processing unit 4555. The functions of each module will be described below.
[0060] In other embodiments, the image processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the image processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.
[0061] The image processing method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.
[0062] See also Figure 3 , Figure 3 is an optional flow chart of the image processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.
[0063] S101, obtaining an image to be processed and screen layout information of the image to be processed; the screen layout information includes a picture shooting direction and / or an image size.
[0064] The image processing method provided in the embodiment of the present application is suitable for performing image segmentation on the image to be processed through a neural network model to obtain an image segmentation result, and then further performing image effect processing based on the segmentation result. For example, a portrait part is segmented from an image through a trained neural network model, and the background part other than the portrait part is blurred, or a specified image part is segmented from an image through a trained neural network model for mosaic occlusion processing, etc. The embodiment of the present application does not limit the specific application scenario.
[0065] In S101 , the image processing apparatus first obtains an image to be processed and screen layout information of the image to be processed.
[0066] In an embodiment of the present application, the screen layout information of the image to be processed may include the picture shooting direction and / or image size of the image to be processed, wherein the picture shooting direction may be obtained by a gravity sensor on an image acquisition device that acquires the image to be processed. Exemplarily, when a mobile phone is performing image acquisition, it may obtain the gravity sensing data of the mobile phone when the mobile phone is currently performing image acquisition through the installed gyroscope sensor, and the gravity sensing data represents the current shooting posture of the mobile phone. The image processing device may analyze the corresponding picture shooting direction according to the current shooting posture of the mobile phone. The image size may be data representing the aspect ratio of the image to be processed, and the image processing device may obtain whether the layout type of the image to be processed belongs to a horizontal type or a vertical type through the aspect ratio of the image to be processed. The image processing device to be processed obtains the screen layout information of the image to be processed while acquiring the image to be processed, for further image processing in the future.
[0067] S102: Determine a target screen direction type of the image to be processed according to screen layout information of the image to be processed.
[0068] In an embodiment of the present application, when the screen layout information of the image to be processed includes the picture shooting direction, the image processing device can directly determine the screen direction of the image to be processed that can be obtained by shooting in the picture shooting direction according to the picture shooting direction, as the target screen direction type. When the screen layout information is the image size, the image processing device can obtain the aspect ratio of the image to be processed according to the image size, and then determine whether the image to be processed is a horizontal picture or a vertical picture. In this way, the image processing device can determine which type of image with an aspect ratio of the image to be processed belongs to according to the screen layout information of the image to be processed, that is, determine the target screen direction type of the image to be processed.
[0069] In an embodiment of the present application, the screen direction type characterizes the aspect ratio of the image to be processed. In some embodiments, the screen direction type can be divided into at least two types according to the aspect ratio. Exemplarily, the screen direction type can be divided into a horizontal type or a vertical type. When the aspect ratio of the image to be processed is 3:4, its screen direction type belongs to the vertical type; when the aspect ratio of the image to be processed is 4:3 or 16:9, its screen direction type belongs to the horizontal type. In other embodiments, the screen direction type can be further divided into a horizontal positive type, a horizontal reverse type, a vertical positive type and a vertical reverse type according to the direction of the main body of the screen; or it can be divided into other types according to the needs of image processing, such as different values of aspect ratios. The specific selection is made according to the actual situation, and the embodiment of the present application is not limited.
[0070] In the embodiment of the present application, the target screen direction type is a screen direction type corresponding to the image to be processed among the at least two screen direction types.
[0071] S103: According to the correspondence between the preset picture direction types and the image segmentation models, a target image segmentation model corresponding to the target picture direction type is determined from at least two preset image segmentation models.
[0072] In an embodiment of the present application, since the target screen direction type can characterize the aspect ratio of the image to be processed, in order to keep the ratio of the image to be processed unchanged during the image processing process, the image processing device will select an image segmentation model that meets the target screen direction type from at least two preset image segmentation models according to the target screen direction type as the target image segmentation model.
[0073] In the embodiment of the present application, the at least two preset image segmentation models can be image segmentation models trained according to sample image sets of at least two screen direction types. When training each image segmentation model, the image processing device uses sample data of the corresponding screen direction type for training, so that the at least two image segmentation models obtained by training correspond to the at least two screen direction types one by one, and the default layout direction of each image segmentation model is the corresponding screen direction type. Therefore, the image processing device can determine the target image segmentation model corresponding to the target screen direction type from the at least two preset image segmentation models according to the correspondence between the preset screen direction type and the image segmentation model.
[0074] In some embodiments of the present application, when at least two screen orientation types can be classified as horizontal and vertical types, the image processing device can use a set of sample images of the horizontal type to train the initial neural network model to obtain a first segmentation model, so that the input image ratio required by the first segmentation model matches the size ratio of the image to be processed itself. Image segmentation of the image to be processed of the horizontal screen orientation type using the first segmentation model can reduce image distortion occurring during the processing process. Therefore, when the target screen orientation type belongs to one of the horizontal screen orientation types, the image processing device can determine the first segmentation model as the target image segmentation model; or, the image processing device can use a set of sample images of the vertical type to train the initial neural network model to obtain a second segmentation model, so that when the target screen orientation type belongs to one of the vertical screen orientation types, the image processing device can determine the second segmentation model as the target image segmentation model.
[0075] In the embodiment of the present application, the image segmentation model can be a CNN network model, or it can be a semantic segmentation network deeplab, a fully convolutional neural network (Fully Convolutional Network, FCN), a densely connected network (Densely Connected Convolutional Networks, DenseNet), a deep residual network (Deep ResNet, DRN), etc. The specific selection is based on the actual situation and is not limited in the embodiment of the present application.
[0076] S104, using the target image segmentation model to perform image segmentation on the image to be processed to obtain an image segmentation result.
[0077] In the embodiment of the present application, the image processing device uses a determined target image segmentation model to perform image segmentation on the preprocessed image to obtain an image segmentation result.
[0078] In the embodiment of the present application, based on the specific purpose of the target image segmentation model, the target image segmentation model can segment portraits or target images of other styles, and then obtain a mask of the corresponding image area of the target image in the image to be processed as the image segmentation result. The specific selection is made according to the actual application, and the embodiment of the present application does not limit it.
[0079] In some embodiments, Figure 8 As shown, when the image to be processed is a portrait photo, the image processing device uses the target image segmentation model to identify and extract the portrait part from the image to be processed 10, and uses the generated portrait mask 20 as the image segmentation result.
[0080] In the embodiment of the present application, the image segmentation algorithm adopted by the target image segmentation model may be a convolutional neural network CNN algorithm, or a deeplab series segmentation algorithm, a deep learning U-Net, FCN, DenseNet, DRN algorithm, etc. For example, when segmenting a portrait, the target image segmentation model may identify the body of a person in the image to be processed as one category, and the background other than the body of the person as another category, and generate a corresponding portrait mask according to the identified boundary of the body of the person as the image processing result.
[0081] S105: Perform image processing based on the image segmentation result to obtain a final processed image.
[0082] In the embodiment of the present application, after the image processing device obtains the image segmentation result through the target image segmentation model, it performs image processing based on the image segmentation result to obtain a final processed image.
[0083] In some embodiments, the image processing device may perform further image effect processing based on the image segmentation result and according to specific image processing requirements to generate a final processed image.
[0084] Exemplarily, when the image segmentation result is a target image and a background image, the image processing device can blur the background image to highlight the target image; or replace the background image with another background. The image processing device can also perform mosaic processing, add decorative effects, beautify processing, etc. on the target image, or perform image effect processing on the target image and the background image at the same time to obtain the final effect image. The specific selection is made according to the actual situation, and the embodiment of the present application is not limited.
[0085] It can be understood that in the embodiment of the present application, the image processing device can select the corresponding target image segmentation model to perform image segmentation operations on the image to be processed according to the target screen direction type of the image to be processed, thereby ensuring that the target screen direction type matches the default layout direction required by the target image segmentation model, ensuring the accuracy of the image segmentation results, and thus improving the accuracy of image processing.
[0086] In some embodiments, see Figure 4 , Figure 4 is an optional flow chart of the method provided in the embodiment of the present application, Figure 3 The illustrated S101 can be implemented through S1011 - S1015 , which will be described in conjunction with each step.
[0087] S1011. Obtain original image and original screen layout information.
[0088] In the embodiment of the present application, the image to be processed may be an image obtained by preprocessing the original image. The image processing device obtains the original image and the original screen layout information, and is used to preprocess the original image according to the original screen layout information to obtain the image to be processed.
[0089] In the embodiment of the present application, the original screen layout information may include the original image size and the original image shooting direction, wherein the original image size may be the width and height information of the original image, and the original image shooting direction may be the shooting direction obtained by analyzing the sensor sensing data on the image acquisition device.
[0090] S1012: Scale the original image to a preset input image size to obtain a scaled image.
[0091] In the embodiment of the present application, at least two image segmentation models correspond to different aspect ratios of the screen direction types, but the preset input picture sizes of the two are consistent, and the preset input picture size is the size of the picture that can be processed by the image segmentation model. The picture processing device scales the original image in equal proportion according to the preset input picture sizes corresponding to the at least two image segmentation models to obtain a scaled image.
[0092] S1013: normalize the scaled image to obtain an image to be processed.
[0093] In an embodiment of the present application, the image processing device normalizes the scaled image after scaling the original image. Exemplarily, the image to be processed can be obtained by first subtracting the mean of each pixel of the original image and then dividing the variance.
[0094] In the embodiment of the present application, the image processing device may also use other normalization algorithms to normalize the scaled image. The specific selection is made according to the actual situation and is not limited in the embodiment of the present application.
[0095] S1014. When the original picture layout information includes the original shooting direction of the image, the original shooting direction of the image is used as the picture layout information of the image to be processed; the original shooting direction of the image is the shooting direction obtained from the sensor of the image acquisition device of the original image.
[0096] In the embodiment of the present application, when the original screen layout information includes the original shooting direction of the image, the image processing device directly uses the original shooting direction of the image as the screen layout information of the image to be processed.
[0097] For example, when a mobile phone is used to take a portrait photo, the original image is the image captured by the mobile phone camera, and the original picture layout information is the shooting direction information obtained by analyzing the sensing data collected by the gyroscope on the mobile phone.
[0098] S1015: When the original picture layout information does not include the original shooting direction of the image, obtain the image size of the image to be processed as the picture layout information.
[0099] In the embodiment of the present application, when the original screen layout information does not include the original shooting direction of the image, after the image processing device obtains the image to be processed, it uses the size of the image to be processed as the screen layout information of the image to be processed.
[0100] Exemplarily, when the original image is obtained by non-shooting methods such as downloading from the network or taking a screenshot, and the original screen layout information does not include shooting direction information, the image processing device uses the size of the image to be processed as the screen layout information.
[0101] It is understandable that in the embodiments of the present application, the image processing device can pre-process the original images acquired in different ways, so as to obtain the image to be processed and perform the next step of image segmentation and image processing, thereby improving the flexibility of image processing.
[0102] In some embodiments, see Figure 5 , Figure 5 is an optional flow chart of the method provided in the embodiment of the present application. When the screen layout information of the image to be processed includes the picture shooting direction, Figure 3 or Figure 4 The illustrated S102 can be implemented through S1021; when the screen layout information of the image to be processed includes the image size, S102 can be implemented through S1022-S1025, which will be described in conjunction with each step.
[0103] S1021. Determine the target screen direction type of the image to be processed according to the image shooting direction of the image to be processed. The image shooting directions include: horizontal positive, horizontal reverse, vertical positive and vertical reverse; and the target screen direction types include: horizontal positive type, horizontal reverse type, vertical positive type and vertical reverse type.
[0104] In the embodiment of the present application, the picture shooting direction corresponds to the target screen direction type one by one, and the image processing device can determine the target screen direction type of the image to be processed according to the picture shooting direction of the image to be processed.
[0105] In some embodiments, images captured by a mobile phone or a camera device with a sensor having a gyroscope-like function may include the following four image shooting directions, namely, the horizontal positive direction as shown in FIG6(a), the horizontal reverse direction as shown in FIG6(b), the vertical positive direction as shown in FIG6(c), and the vertical reverse direction as shown in FIG6(d).
[0106] It can be understood that in the embodiment of the present application, when the screen layout information of the image to be processed includes the picture shooting direction, the image processing device can directly determine the target screen direction type of the image to be processed according to the picture shooting direction.
[0107] S1022. Determine the primary screen orientation type of the image to be processed according to the image size; the primary screen orientation type includes: a horizontal type and a vertical type.
[0108] In the embodiment of the present application, when the screen layout information is the image size of the image to be processed, the image processing device can determine the primary screen orientation type of the image to be processed according to the aspect ratio represented by the image size: including a horizontal type and a vertical type.
[0109] S1023, performing orientation recognition on the image to be processed to obtain an orientation recognition result; the orientation recognition result includes a forward direction and a reverse direction.
[0110] In the embodiment of the present application, the image processing device may perform orientation recognition on the image to be processed through a feature recognition algorithm to obtain orientation recognition results: including forward and reverse orientations.
[0111] Exemplarily, the image processing device can identify the portrait in the image to be processed through a feature recognition algorithm, and then identify representative feature sub-images from the portrait, such as a head sub-image, a body sub-image, a facial feature sub-image, etc., and then identify the front and back of the portrait in the image to be processed according to the relative display positions of the above feature sub-images in the image to be processed, and then determine whether the orientation of the image to be processed is forward or reverse, and obtain an orientation recognition result. Exemplarily, if the display position of the head sub-image is below the body sub-image, it means that the portrait is reversed, so the image processing device can determine that the orientation recognition result of the image to be processed where the portrait is located is reversed.
[0112] S1024. Determine a secondary screen direction type according to the direction recognition result and the primary screen direction type; the secondary screen direction type includes a horizontal positive type, a horizontal negative type, a vertical positive type, and a vertical negative type.
[0113] S1025: Use the secondary screen direction type as the target screen direction type.
[0114] In the embodiment of the present application, after the image processing device obtains the primary screen direction type of horizontal or vertical, and the recognition result of forward or reverse, the secondary screen direction type can be determined according to the recognition result and the primary screen direction type, including the horizontal forward type, the horizontal reverse type, the vertical forward type, and the vertical reverse type. The image processing device uses the secondary screen direction type as the target screen direction type.
[0115] It can be understood that in an embodiment of the present application, when the screen layout information of the image to be processed includes the image size, the image processing device can determine the screen direction type of the image to be processed based on the image size and the orientation recognition method, and then select the corresponding image segmentation model for image processing, thereby improving the accuracy of image processing.
[0116] In some embodiments, see Figure 7 , Figure 7 is an optional flowchart of the image processing method provided in the embodiment of the present application, Figure 3 The illustrated S103 can be implemented through S1031 - S1032 , which will be described in conjunction with each step.
[0117] S1031. When the target screen direction type is a horizontal forward type or a horizontal reverse type, determine that the target image segmentation model is a first segmentation model; the first segmentation model is a model for performing image segmentation on an image to be processed whose width is greater than its height.
[0118] In the embodiment of the present application, the first segmentation model is a model for performing image segmentation on an image to be processed whose width is greater than its height, and the first segmentation model can be obtained by training an initial neural network using a set of sample images of a horizontal positive type. When the target screen direction type is a horizontal positive type or a horizontal reverse type, in order to keep the image ratio of the image to be processed undistorted during the image processing process, the image processing device determines that the target image segmentation model is the first segmentation model.
[0119] For a transversely inverted image to be processed, the image processing device may adjust the direction of the transversely inverted image to be processed before inputting the image into the first segmentation model, which will be described in detail later.
[0120] S1032. When the target screen direction type is a vertical forward type or a vertical reverse type, determine that the target image segmentation model is a second segmentation model; the second segmentation model is a model for performing image segmentation on an image to be processed whose width is smaller than its height.
[0121] In the embodiment of the present application, the second segmentation model is a model for performing image segmentation on an image to be processed whose width is smaller than its height, and the second segmentation model can be obtained by training an initial neural network using a set of sample images of a longitudinal positive type. When the target screen direction type is a longitudinal positive type or a longitudinal reverse type, in order to keep the image ratio of the image to be processed undistorted during the image processing process, the image processing device determines that the target image segmentation model is the second segmentation model.
[0122] For a longitudinally inverted image to be processed, the image processing device may adjust the direction of the longitudinally inverted image to be processed before inputting the image into the second segmentation model, which will be described in detail later.
[0123] It can be understood that in the embodiment of the present application, the image processing device can select different image segmentation models to perform image segmentation processing according to the aspect ratio of the image to be processed, thereby avoiding the distortion caused by the inconsistency between the image ratio to be processed by the image segmentation model and the image ratio to be processed, thereby improving the accuracy of image processing.
[0124] In some embodiments, Figure 3 The illustrated S104 can be implemented through S1041 - S1042 , which will be described in conjunction with each step.
[0125] S1041. Rotate the image to be processed according to a preset input image direction of the target image segmentation model to obtain an adjusted image to be processed.
[0126] In the embodiment of the present application, after the image processing device determines the target image segmentation model, it can adjust the direction of the image to be processed according to the preset input picture direction of the target image segmentation model to obtain the adjusted image to be processed.
[0127] In the embodiment of the present application, the preset input image direction is the default image direction of the input image required by the target image segmentation model, that is, the image direction of the sample image used by the target image segmentation model in the training stage. In order to ensure that the current image segmentation module can perform correct image segmentation processing on the image to be processed, the image processing device needs to adjust the direction of the image to be processed according to the preset input image direction so that the directions of the two are consistent, thereby obtaining the adjusted image to be processed.
[0128] In some embodiments, when the screen layout information of the image to be processed includes the picture shooting direction, S1041 can be implemented through S201-S02, which will be described in combination with each step:
[0129] S201, calculating the angle between the image shooting direction and the preset input image direction.
[0130] S202 , rotating and adjusting the image to be processed according to the angle, so that the image direction of the rotated image to be processed is consistent with the preset input image direction, thereby obtaining an adjusted image to be processed.
[0131] In an embodiment of the present application, the image processing device rotates and adjusts the image to be processed according to the angle difference between the image shooting direction and the preset input image, so that the direction of the rotated image to be processed is consistent with the direction of the preset input image, and uses the rotated and adjusted image to be processed as the adjusted image to be processed.
[0132] In some embodiments, the preset input image direction of the target image segmentation model corresponding to the horizontal type screen direction type can be a horizontal positive direction. For the image to be processed whose picture shooting direction is a horizontal reverse direction, the image processing device can rotate it 180 degrees clockwise or counterclockwise to make the adjusted picture shooting direction of the image to be processed consistent with the preset input image direction; and for the image to be processed whose picture shooting direction is a horizontal positive direction, the image processing device does not need to rotate the image to be processed and can directly use it as the adjusted image to be processed.
[0133] It should be noted that, in the embodiment of the present application, when the layout information of the screen to be processed is the image size, the image processing device can use the recognition result of the orientation recognition as the picture shooting direction of the image to be processed, and process the image to be processed through S201-S202.
[0134] S1042: Based on the target image segmentation model, identify and extract the image region to be processed from the adjusted image to be processed as the image segmentation result.
[0135] In an embodiment of the present application, the image processing device inputs the adjusted image to be processed into the target image segmentation model, uses the target image segmentation model, and identifies the target image from the adjusted image to be processed through inference of the network model, and extracts the area where the target image is located as the image area to be processed, as the image segmentation result.
[0136] In the embodiment of the present application, the image processing device uses the target image segmentation model to identify and extract the target image from the adjusted image to be processed. The description of the implementation process of the image segmentation result is consistent with S103 and will not be repeated here.
[0137] It can be understood that in an embodiment of the present application, when the screen layout information of the image to be processed includes the picture shooting direction, the image processing device can directly determine the screen direction type of the image to be processed according to the picture shooting direction, and then select the corresponding image segmentation model for image processing, thereby improving the accuracy of image processing.
[0138] In some embodiments, Figure 3 The illustrated S105 can be implemented through S1051, which will be described in conjunction with each step.
[0139] S1051. Based on the image segmentation result, perform image processing on the image area to be processed to obtain a final processed image.
[0140] In the embodiment of the present application, after the image processing device obtains the image segmentation result, it obtains the image region to be processed in the image to be processed. The image processing device performs image processing on the image region to be processed to obtain a final processed image.
[0141] In some embodiments, see Fig. 9 , Fig. 9 is an optional flow chart of the method provided in the embodiment of the present application, Figure 3 Before S103 shown, S301 to S303 may also be included, and each step will be described in combination.
[0142] S301, obtaining a training sample set; the training sample set includes a plurality of training sample objects, and the training sample objects correspond to sample images and label images corresponding to the sample objects.
[0143] In an embodiment of the present application, an image processing device may train a sample set, the training sample set includes a plurality of training sample objects, each training sample object includes a sample image and a label image corresponding to the sample object. The sample image may be an original training image, and the label image may be an image segmentation result corresponding to the original training image, such as a portrait mask. The image directions in the sample image and the label image are the preset input image directions corresponding to the initial first segmentation model or the preset input image directions corresponding to the initial second segmentation model, such as horizontal positive or vertical positive, and the image type may be a raw type, a YUV type, an RGB type, a depth map type, etc., which are not limited in the embodiment of the present application.
[0144] S302: Perform image ratio preprocessing on the training sample set according to the first preset input ratio of the initial first segmentation model and the second preset input ratio of the initial second segmentation model to obtain the first training sample set and the second training sample set.
[0145] In some embodiments, Fig. 9 The illustrated S302 can be implemented through S3021 - S3022 , which will be described in conjunction with each step.
[0146] S3021. When training an initial first segmentation model, perform image preprocessing on a training sample set according to a first preset input ratio to obtain a first training sample set; the first training sample set includes multiple first sample objects; the first sample object includes a first sample image and a first label image corresponding to the first sample image.
[0147] In an embodiment of the present application, when training the initial first segmentation model, for a training sample object in the training sample set, when the size ratio of the sample image in the training sample object and its corresponding label image do not conform to the first preset input ratio, exemplarily, when the sample image and the label image are vertical type pictures, the image processing device will perform edge fill on the sample image and its corresponding label image according to the first preset input ratio to obtain a first filled sample image and a first filled label image.
[0148] In the embodiment of the present application, the image processing device can determine whether the size ratio of the sample image and the label image is consistent with the first preset input ratio according to the width and height of the sample image and the label image.
[0149] Exemplarily, when the first preset input ratio of the initial first segmentation model is of the horizontal type, the edges of the sample image and the label image of the vertical type need to be filled. When filling, the long side of the sample image of the vertical type remains unchanged, and the short side is filled to 4 / 3, 16 / 9 or other sizes of the long side. If the first preset input ratio is 4:3 and the sample image ratio is 3:4, the short side of the sample image is filled to 4 / 3 of the long side. The label image can also be filled in this way. When the sample image is an RGB image, the filling values of the three channels are all 127.5, or the average of the three channels of the sample image is used as the filling value of the sample image. The filling value of the label image is the "ignore category" value, which is usually 255.
[0150] In the embodiment of the present application, after obtaining the first filled sample image and the first filled label image, the image processing device scales and crops the first filled sample image and the first filled label image according to the preset input picture size of the initial first segmentation model, such as 640*480, to obtain the first scaled sample image and the first scaled label image. It should be noted that when the size ratio of the sample image in the training sample object and its corresponding label image is consistent with the first preset input ratio, the image processing device can directly scale and crop the sample image and the label image to obtain the first scaled sample image and the first scaled label image.
[0151] In the embodiment of the present application, the image processing device uses the first scaled label image as the first label image, and then performs normalization processing on the first scaled sample image to effectively converge the neural network and prevent the network from overfitting, thereby obtaining the first sample image. The image processing device performs the same processing on each training sample object in the training sample set, thereby obtaining multiple first sample objects composed of multiple first sample images and corresponding first label images, and the image processing device uses the multiple first sample objects as the first training sample set.
[0152] S3022. When training the initial second segmentation model, perform image preprocessing on the training sample set according to a second preset input ratio to obtain a second training sample set; the second training sample set includes multiple second sample objects; the second sample objects include a second sample image and a second label image corresponding to the second sample image.
[0153] In the embodiment of the present application, when training the initial second segmentation model, the image processing device performs image preprocessing on the sample image set according to the second preset input ratio, and the process of obtaining the second sample image set is consistent with the description of S3021 and will not be repeated here.
[0154] S303, based on the first training sample set and the second training sample set, continuously train the initial first segmentation model and the initial second segmentation model respectively, and iteratively update the network model parameters of the initial first segmentation model and the initial second segmentation model through a preset cost function until the preset training conditions are met, thereby obtaining the first segmentation model and the second segmentation model.
[0155] In an embodiment of the present application, the image processing device continuously trains the initial first segmentation model and the initial second segmentation model based on the first training sample set and the second training sample set, respectively. Exemplarily, during the training process of the initial first segmentation model, the image processing device performs convolutional neural network training on the initial first segmentation model based on the first training sample set, and uses the cost function to perform back propagation to update the model parameters. Iterate continuously until the network meets the preset training conditions, the iteration ends, and the first segmentation model is obtained. During the training process of the initial second segmentation model, the image processing device performs convolutional neural network training on the initial second segmentation model based on the second training sample set, and uses the cost function to perform back propagation to update the model parameters. Iterate continuously until the network meets the preset training conditions, the iteration ends, and the second segmentation model is obtained.
[0156] In an embodiment of the present application, the preset training condition may be that the number of training times reaches a preset number, or the error of the image segmentation result output by the segmentation network model reaches a convergence condition, etc. The specific selection is made based on actual conditions and is not limited in the embodiment of the present application.
[0157] In the present application embodiment, Fig.10 As shown, the first training sample set includes positive samples and negative samples obtained according to a preset configuration ratio. The positive sample is a positive sample image and a corresponding positive sample to-be-processed area, and the negative sample is a negative sample image and a corresponding negative sample to-be-processed area. Exemplarily, when segmenting a portrait, the positive sample image may be a sample image containing a portrait, and the positive sample to-be-processed area may be a portrait area in the positive sample image; the negative sample image may be a sample image not containing a portrait, and the negative sample to-be-processed area may be an area of a background image in the negative sample image. The image processing device uses the first training sample set and based on the cost function, calls the initial first segmentation model to process positive samples or negative samples to obtain a first training result; the image processing device continuously detects the first segmentation model in training until the first training result meets the preset condition, ends the training, and obtains the first segmentation model. The preset condition is used to characterize the image segmentation result obtained according to the first segmentation model in training that is closest to the real segmented image part.
[0158] In the embodiment of the present application, the cost function can be used to evaluate the quality of the model. The smaller the cost function, the more consistent the model and parameters are with the training samples. In some embodiments, the image processing device can use a cost function such as a normalized exponential function softmax or cross entropy to punish and back-propagate to update the convolutional neural network parameters, and iterate continuously until the network meets the convergence conditions and the iteration ends.
[0159] In the embodiment of the present application, the iterative training process of the initial second segmentation model by the image processing device is consistent with the description of the iterative training process of the initial first segmentation model, which will not be repeated here.
[0160] It can be understood that in the embodiment of the present application, the image processing device can use a set of sample images of a horizontal type and a set of labeled images to train a first segmentation model, and use a set of sample images of a vertical type and a set of labeled images to train a second segmentation model, so that the image processing device can respectively select the network model corresponding to the screen direction type of the image to be processed to perform image segmentation operations when performing image processing, thereby improving the accuracy of image processing. In the embodiment of the present application, when the image to be processed is an image obtained by preprocessing the original image by scaling and cropping, the image processing device uses the target image segmentation model to perform image segmentation on the image to be processed. After obtaining the image segmentation result, the image segmentation result can be restored to the original image size first, so as to obtain the image segmentation result in the original image. The corresponding image area to be processed, and then the image processing device performs image processing on the image area to obtain the final processed image.
[0161] In some embodiments, see Fig.11 The image processing device can perform image segmentation on the image to be processed 11 by the method in S101-S104 to obtain an image segmentation result, namely, a portrait mask 21. The image processing device can first restore the portrait mask 21 to the size of the original image, obtain the area where the portrait is located in the original image based on the portrait mask 21, and blur the background area of the portrait to obtain the image as shown in FIG. Fig.11 The background blur effect image 31 is shown.
[0162] It can be understood that in the embodiment of the present application, the image processing device can perform further effect processing on the image based on the image segmentation result. Since the image segmentation result is obtained through an image segmentation model corresponding to the picture direction type of the image to be processed, the accuracy of the image segmentation result is guaranteed, thereby also improving the accuracy of image processing based on the image segmentation result.
[0163] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.
[0164] In some embodiments, the initial first segmentation model is an initial horizontal shooting model. The method for training the horizontal shooting model provided in the embodiment of the present application can be shown in FIG. 12( a ):
[0165] S401, obtaining the original training image and label mask.
[0166] In the embodiment of the present application, the training original image is a sample image in the training sample object, and the label mask is a label image corresponding to the sample image. The process of S401 is consistent with the description of S301, and will not be repeated here.
[0167] S402: Determine whether the original training image and the label mask meet a first preset input ratio.
[0168] In the embodiment of the present application, the image processing device determines whether the training original image and the label mask meet the first preset input ratio. If they do not meet the first preset input ratio, S403 is executed; if they meet the first preset input ratio, the execution is jumped to S404.
[0169] S403: Fill the edges of the original training image and the label mask that do not conform to the first preset input ratio.
[0170] In the embodiment of the present application, the image processing device performs edge filling on the training original image and the label mask that do not conform to the first preset input ratio to obtain a first filled sample image and a first filled label image.
[0171] S404: scaling and cropping.
[0172] In an embodiment of the present application, the image processing device scales and crops the training original image and label mask that meet the first preset input ratio obtained by S402, and the first filled sample image and the first filled label image obtained by S403 to the preset input image size required by the initial horizontal shooting model, to obtain the first scaled sample image and the first scaled label image.
[0173] In the embodiment of the present application, the first zoomed label image is the first label image.
[0174] S405: Normalize the first scaled sample image.
[0175] In the embodiment of the present application, the image processing device performs a normalization operation on the first scaled sample image to obtain a first sample image.
[0176] In the embodiment of the present application, the process of S402-S405 is consistent with the description of S3021 and will not be repeated here.
[0177] S406: Initial horizontal shooting model training.
[0178] S407: Output the portrait mask.
[0179] In the embodiment of the present application, the image processing device inputs the first sample image into the initial horizontal shooting model, performs image segmentation on the first sample image through the initial horizontal shooting model, and outputs a portrait mask as the image segmentation result of the current training.
[0180] S408: Calculate cost function.
[0181] In an embodiment of the present application, the image processing device performs back propagation calculation of the cost function based on the portrait mask and the first scaled label image outputted from each round of training to obtain adjustment values of the network model parameters, updates the network model parameters of the initial horizontal shooting model based on the adjustment values of the network model parameters, and iterates the processes in S406-S408 until the preset training conditions are met to obtain the horizontal shooting model.
[0182] In the embodiment of the present application, the horizontal shooting model is the first segmentation model.
[0183] In the embodiment of the present application, the process of S406-S408 is consistent with the description of the training process of the initial first segmentation model in S303, and will not be repeated here.
[0184] In some embodiments, the initial second segmentation model is an initial vertical shooting model. The method for training the vertical shooting model provided in the embodiment of the present application can be shown in FIG. 12( b ):
[0185] S501, obtaining the original training image and label mask.
[0186] In the embodiment of the present application, the process of S501 is consistent with the description of S401 and will not be repeated here.
[0187] S502: Determine whether the original training image and the label mask conform to a second preset input ratio.
[0188] In the embodiment of the present application, the image processing device determines whether the training original image and the label mask meet the second preset input ratio. If they do not meet the second preset input ratio, S503 is executed; if they meet the second preset input ratio, the execution jumps to S504.
[0189] S503: Fill the edges of the training original image and the label mask that do not conform to the second preset input ratio.
[0190] In the embodiment of the present application, the image processing device performs edge filling on the training original image and the label mask that do not conform to the second preset input ratio to obtain a second filled sample image and a second filled label image.
[0191] S504: scaling and cropping.
[0192] In an embodiment of the present application, the image processing device scales and crops the training original image and training mask that meet the second preset input ratio obtained by S502, and the second filled sample image and the second filled label image obtained by S503 to the preset input image size required by the initial vertical shooting model, to obtain a second scaled sample image and a second scaled label image.
[0193] In the embodiment of the present application, the processed mask is the second label image.
[0194] S505: Normalize the second scaled sample image.
[0195] In the embodiment of the present application, the image processing device performs a normalization operation on the second scaled sample image to obtain a second sample image.
[0196] In the embodiment of the present application, the process of S502-S505 is consistent with the description of S3022 and will not be repeated here.
[0197] S506: Initial vertical shooting model training.
[0198] S507: Output the portrait mask.
[0199] In the embodiment of the present application, the image processing device inputs the second sample image into the initial vertical shooting model, performs image segmentation on the second sample image through the initial vertical shooting model, and outputs a portrait mask as the image segmentation result of the current training.
[0200] S508: Calculate cost function.
[0201] In an embodiment of the present application, the image processing device performs back propagation calculation through a cost function based on the portrait mask and the second scaled label image outputted in each round of training to obtain adjustment values of the network model parameters, updates the network model parameters of the initial vertical shooting model based on the adjustment values of the network model parameters, and iterates the processes in S506-S508 until the preset training conditions are met to obtain the vertical shooting model.
[0202] In the embodiment of the present application, the vertical shooting model is the second segmentation model.
[0203] In the embodiment of the present application, the process of S506-S508 is consistent with the description of the training process of the initial second segmentation model in S303, and will not be repeated here.
[0204] It can be understood that in the embodiment of the present application, the image processing device can use the training original images and label masks that conform to different preset input ratios to respectively train the horizontal shooting model and the vertical shooting model, so that the image processing device can respectively select the network model corresponding to the preset input ratio to perform image segmentation operations when performing image processing, thereby improving the accuracy of image processing.
[0205] In some embodiments, based on the horizontal shooting model and the vertical shooting model trained in FIG. 12(a) and FIG. 12(b), the preset input image direction of the horizontal shooting model can be a horizontal positive direction, and when the preset input image direction of the vertical shooting model can be a vertical positive direction, then for the scene where the image segmentation result is a portrait mask, the image processing method provided in the embodiment of the present application can be Fig.13 As shown:
[0206] S601, obtaining an original image and a photographing direction.
[0207] In the embodiment of the present application, the photographing direction is the original photographing direction of the image, and the process description of S601 is consistent with that of S1011, which will not be repeated here.
[0208] S602: Scale the original image.
[0209] In the embodiment of the present application, the image processing device scales the original image to obtain a scaled image.
[0210] In the embodiment of the present application, the process of S602 is consistent with the description of S1012 and will not be repeated here.
[0211] S603: normalize the scaled image to obtain an image to be processed.
[0212] In the embodiment of the present application, the process of S603 is consistent with the description of S1013 and will not be repeated here.
[0213] S604: Determine whether to rotate the image to be processed according to the photographing direction.
[0214] In the embodiment of the present application, the image processing device determines whether to rotate the image to be processed according to the shooting direction. If the shooting direction is horizontal positive, it means that the shooting direction is consistent with the preset input image direction of the horizontal shooting model, and jump to S605; if the shooting direction is horizontal reverse, jump to S606; if the shooting direction is vertical positive, it means that the shooting direction is consistent with the preset input image direction of the vertical shooting model, and jump to S607; if the shooting direction is vertical reverse, jump to S608.
[0215] S605: directly input the image to be processed with the shooting direction being the horizontal positive direction into the horizontal shooting model.
[0216] In the embodiment of the present application, the horizontal shooting model is the first segmentation model.
[0217] S606: Rotate the image to be processed, which is photographed in the horizontal direction, by 180 degrees clockwise or counterclockwise and input it into the horizontal shooting model.
[0218] In the embodiment of the present application, the description of the rotation process of S606 is consistent with that of S1041 and will not be repeated here.
[0219] S607: directly input the image to be processed with the shooting direction being the vertical positive direction into the vertical shooting model.
[0220] In the embodiment of the present application, the vertical shooting model is the second segmentation model.
[0221] S608: Rotate the image to be processed, which is in the reverse direction of the vertical direction, by 180 degrees clockwise or counterclockwise, and input it into the vertical shooting model.
[0222] In the embodiment of the present application, the description of the rotation process of S608 is consistent with that of S1041 and will not be repeated here.
[0223] S609: Output the portrait mask.
[0224] In the embodiment of the present application, the image processing device performs network inference on the image to be processed or the rotated image to be processed through the corresponding horizontal shooting model or vertical shooting model, and outputs a portrait mask.
[0225] In the embodiment of the present application, the portrait mask is the image segmentation result.
[0226] In the embodiment of the present application, the process of S609 is consistent with the description of S1042 and will not be repeated here.
[0227] It can be understood that in the embodiment of the present application, the image processing device makes a judgment based on the shooting direction of the original image, selects the corresponding horizontal shooting model or vertical shooting model to perform image segmentation preprocessing and network model inference operations on the original image, and can maintain the aspect ratio of the original image during the preprocessing and network inference process, thereby ensuring the accuracy of the image segmentation result, and ultimately improving the accuracy of image processing.
[0228] The following is a description of an exemplary structure of the image processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the image processing device 455 of the memory 450 may include: an acquisition unit 4551, a determination unit 4552, a selection unit 4553, a segmentation unit 4554 and a processing unit 4555, wherein:
[0229] The acquisition unit 4551 is used to acquire the image to be processed and the screen layout information of the image to be processed; the screen layout information includes the picture shooting direction and the image size;
[0230] A determination unit 4552 is used to determine a target screen direction type of the image to be processed according to the screen layout information of the image to be processed;
[0231] A selection unit 4553 is used to determine a target image segmentation model corresponding to the target picture direction type from at least two preset image segmentation models according to a correspondence between preset picture direction types and image segmentation models;
[0232] The segmentation unit 4554 is used to perform image segmentation on the image to be processed by using the target image segmentation model to obtain an image segmentation result;
[0233] The processing unit 4555 is used to perform image processing based on the image segmentation result to obtain a final processed image.
[0234] In some embodiments of the present application, the segmentation unit 4554 is also used to rotate the image to be processed according to a preset input picture direction of the target image segmentation model to obtain an adjusted image to be processed; based on the target image segmentation model, identify and extract the image area to be processed from the adjusted image to be processed to obtain the image segmentation result.
[0235] In some embodiments of the present application, the screen layout information of the image to be processed includes: the picture shooting direction; the segmentation unit 4554 is also used to calculate the angle between the picture shooting direction and the preset input picture direction; the image to be processed is rotated according to the angle so that the picture direction of the rotated image to be processed is consistent with the preset input picture direction, thereby obtaining the adjusted image to be processed.
[0236] In some embodiments of the present application, the determination unit 4552 is also used to determine the target screen direction type of the image to be processed according to the picture shooting direction of the image to be processed; wherein the picture shooting direction includes: horizontal positive, horizontal reverse, vertical positive and vertical reverse; the target screen direction type includes: horizontal positive type, horizontal reverse type, vertical positive type and vertical reverse type.
[0237] In some embodiments of the present application, the screen layout information of the image to be processed includes: image size; the at least two screen direction types include a primary screen direction type and a secondary screen direction type; the determination unit 4552 is also used to determine the primary screen direction type of the image to be processed according to the image size; the primary screen direction type includes: a horizontal type and a vertical type; the orientation of the image to be processed is identified to obtain an orientation identification result; the orientation identification result includes forward and reverse; based on the orientation identification result and the primary screen direction type, the secondary screen direction type is determined; the secondary screen direction type includes a horizontal forward type, a horizontal reverse type, a vertical forward type and a vertical reverse type; the secondary screen direction type is used as the target screen direction type.
[0238] In some embodiments of the present application, the determination unit 4552 is also used to determine that the preset input image direction is horizontal positive when the screen direction type corresponding to the target image segmentation model is a horizontal positive type or a horizontal reverse type; and to determine that the preset input image direction is vertical positive when the screen direction type corresponding to the target image segmentation model is a vertical positive type or a vertical reverse type.
[0239] In some embodiments of the present application, the acquisition unit 4551 is also used to acquire the original image and original screen layout information; scale the original image to a preset input image size to obtain a scaled image; normalize the scaled image to obtain the image to be processed; when the original screen layout information includes the original shooting direction of the image, the original shooting direction of the image is used as the screen layout information of the image to be processed; the original shooting direction of the image is the shooting direction obtained from the sensor of the image acquisition device of the original image; when the original screen layout information does not include the original shooting direction of the image, the image size of the image to be processed is obtained as the screen layout information.
[0240] In some embodiments of the present application, the at least two image segmentation models include: a first segmentation model and a second segmentation model; the determination unit 4552 is also used to determine that the target image segmentation model is the first segmentation model when the target screen direction type is the horizontal forward type or the horizontal reverse type; the first segmentation model is a model for performing image segmentation on the image to be processed whose width is greater than the height; when the target screen direction type is the vertical forward type or the vertical reverse type, the target image segmentation model is determined to be the second segmentation model; the second segmentation model is a model for performing image segmentation on the image to be processed whose width is less than the height.
[0241] In some embodiments of the present application, the image processing device 455 also includes a training unit, which is used to obtain a training sample set; the training sample set includes multiple training sample objects, and the training sample objects correspond to sample images and label images corresponding to the sample images; according to the first preset input ratio of the initial first segmentation model and the second preset input ratio of the initial second segmentation model, the training sample set is image ratio preprocessed to obtain a first training sample set and a second training sample set; based on the first training sample set and the second training sample set, the initial first segmentation model and the initial second segmentation model are continuously trained respectively, and the model network parameters of the initial first segmentation model and the initial second segmentation model are iteratively updated through a preset cost function until the preset training conditions are met, so as to obtain the first segmentation model and the second segmentation model.
[0242] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.
[0243] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned image processing method of the embodiment of the present application.
[0244] It should be noted that in the embodiment of the present application, if the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable the terminal (which can be a smart phone with a camera, a tablet computer, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.
[0245] The present application embodiment provides a computer-readable storage medium storing a computer program, wherein the computer program is stored. When the computer program is executed by a processor, the processor will be caused to execute the method provided by the present application embodiment, for example, Figure 3 , 4 , the methods shown in 5, 7, 9, 12, and 13.
[0246] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.
[0247] In some embodiments, a computer program may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0248] As an example, a computer program may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).
[0249] By way of example, a computer program may be deployed to be executed on one computing device or on multiple computing devices at one site or on multiple computing devices distributed across multiple sites and interconnected by a communication network.
[0250] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.
[0251] To summarize, through the embodiments of the present application, it is possible to select a corresponding image segmentation model to perform image segmentation operations on the image to be processed according to the picture direction type of the image to be processed, thereby maintaining the original image ratio of the image to be processed during image preprocessing according to the preset size of the model and the model inference process, thereby ensuring the accuracy of the image segmentation results and improving the accuracy of image processing.
[0252] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. An image processing method, characterized in that: include: Acquire an image to be processed and screen layout information of the image to be processed; The picture layout information includes the picture shooting direction and / or the image size; Determining a target screen direction type of the image to be processed according to the screen layout information of the image to be processed; According to the correspondence between the preset picture direction type and the image segmentation model, a target image segmentation model corresponding to the target picture direction type is determined from at least two preset image segmentation models; the at least two image segmentation models include: a first segmentation model and a second segmentation model; Using the target image segmentation model to perform image segmentation on the image to be processed to obtain an image segmentation result; Performing image processing based on the image segmentation result to obtain a final processed image; The step of determining the target screen direction type of the image to be processed according to the screen layout information of the image to be processed includes: Determine the target screen direction type of the image to be processed according to the picture shooting direction of the image to be processed; wherein the picture shooting direction includes: horizontal positive, horizontal reverse, vertical positive and vertical reverse; the target screen direction types include: horizontal positive type, horizontal reverse type, vertical positive type and vertical reverse type; Wherein, determining a target image segmentation model corresponding to the target picture direction type from at least two preset image segmentation models according to the correspondence between the preset picture direction type and the image segmentation model comprises: When the target screen direction type is the horizontal forward type or the horizontal reverse type, determining that the target image segmentation model is the first segmentation model; the first segmentation model is a model for performing image segmentation on an image to be processed whose width is greater than its height; When the target screen direction type is the longitudinal forward type or the longitudinal reverse type, the target image segmentation model is determined to be the second segmentation model; the second segmentation model is a model for performing image segmentation on an image to be processed whose width is smaller than its height.
2. The method according to claim 1, characterized in that The step of using the target image segmentation model to perform image segmentation on the image to be processed to obtain an image segmentation result includes: Rotating the image to be processed according to a preset input image direction of the target image segmentation model to obtain an adjusted image to be processed; Based on the target image segmentation model, the image region to be processed is identified and extracted from the adjusted image to be processed to obtain the image segmentation result.
3. The method according to claim 2, characterized in that The picture layout information of the image to be processed includes: the picture shooting direction; The step of rotating the image to be processed according to a preset input image direction of the target image segmentation model to obtain an adjusted image to be processed includes: Calculating the angle between the image shooting direction and the preset input image direction; The image to be processed is rotated according to the angle so that the image direction of the rotated image to be processed is consistent with the preset input image direction, thereby obtaining the adjusted image to be processed.
4. The method according to claim 1, characterized in that: The screen layout information of the image to be processed includes: image size; at least two screen direction types including a primary screen direction type and a secondary screen direction type; The step of determining the target screen direction type of the image to be processed according to the screen layout information of the image to be processed includes: Determine the primary screen orientation type of the image to be processed according to the image size; the primary screen orientation type includes: a horizontal type and a vertical type; Performing orientation recognition on the image to be processed to obtain an orientation recognition result; the orientation recognition result includes a forward direction and a reverse direction; Determining the secondary screen direction type according to the direction recognition result and the primary screen direction type; the secondary screen direction type includes a horizontal positive type, a horizontal reverse type, a vertical positive type, and a vertical reverse type; The secondary screen direction type is used as the target screen direction type.
5. The method according to claim 3, characterized in that: The method further comprises: When the screen direction type corresponding to the target image segmentation model is a horizontal positive type or a horizontal negative type, determining that the preset input picture direction is a horizontal positive direction; When the screen direction type corresponding to the target image segmentation model is a vertical positive type or a vertical negative type, it is determined that the preset input picture direction is a vertical positive direction.
6. The method according to claim 1, characterized in that The obtaining of the image to be processed and the screen layout information of the image to be processed includes: Obtaining original image and original screen layout information; Scaling the original image to a preset input image size to obtain a scaled image; Normalizing the scaled image to obtain the image to be processed; When the original picture layout information includes the original shooting direction of the image, the original shooting direction of the image is used as the picture layout information of the image to be processed; the original shooting direction of the image is the shooting direction obtained from the sensor of the image acquisition device of the original image; When the original picture layout information does not include the original shooting direction of the image, the image size of the image to be processed is obtained as the picture layout information.
7. The method according to claim 1, characterized in that The method further comprises: Acquire a training sample set; the training sample set includes a plurality of training sample objects, and the training sample objects correspond to sample images and label images corresponding to the sample images; According to a first preset input ratio of an initial first segmentation model and a second preset input ratio of an initial second segmentation model, respectively performing image ratio preprocessing on the training sample set to obtain a first training sample set and a second training sample set; Based on the first training sample set and the second training sample set, the initial first segmentation model and the initial second segmentation model are continuously trained respectively, and the model network parameters of the initial first segmentation model and the initial second segmentation model are iteratively updated through a preset cost function until the preset training conditions are met, thereby obtaining the first segmentation model and the second segmentation model.
8. An image processing device, characterized in that: include: An acquisition unit, used for acquiring an image to be processed and screen layout information of the image to be processed; The picture layout information includes the picture shooting direction and / or the image size; A determination unit, configured to determine a target screen direction type of the image to be processed according to screen layout information of the image to be processed; A selection unit, configured to determine a target image segmentation model corresponding to a target picture direction type from at least two preset image segmentation models according to a correspondence between preset picture direction types and image segmentation models; the at least two image segmentation models include: a first segmentation model and a second segmentation model; A segmentation unit, used to perform image segmentation on the image to be processed using the target image segmentation model to obtain an image segmentation result; A processing unit, used for performing image processing based on the image segmentation result to obtain a final processed image; The determination unit is further used to determine the target screen direction type of the image to be processed according to the picture shooting direction of the image to be processed; wherein the picture shooting direction includes: horizontal positive, horizontal reverse, vertical positive and vertical reverse; the target screen direction type includes: horizontal positive type, horizontal reverse type, vertical positive type and vertical reverse type; The determination unit is further used to determine that the target image segmentation model is the first segmentation model when the target screen direction type is the horizontal forward type or the horizontal reverse type; the first segmentation model is a model for performing image segmentation on the image to be processed whose width is greater than its height; when the target screen direction type is the vertical forward type or the vertical reverse type, determine that the target image segmentation model is the second segmentation model; the second segmentation model is a model for performing image segmentation on the image to be processed whose width is less than its height.
9. An image processing device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing the computer program stored in the memory.
10. A computer-readable storage medium, characterized in that: A computer program is stored, which is used to cause a processor to execute and implement the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Picture segment display method and client-side
CN103686064A
Seal removing method
CN109886974A
Image segmentation method and device and terminal equipment
CN110084828A
Image segmentation method and device, electronic equipment and storage medium
CN110189340A
Image segmentation method, device, computer readable storage medium and computer equipment
CN110363780A
