A camera shooting picture quality optimization method and camera

Abstract

The present invention proposes a method for optimizing the quality of camera pictures taken and a camera. The method performs object recognition on the camera pictures taken, divides the pictures taken into a plurality of object partitions according to the object recognition results, so that each object partition contains a key object, the key object is an object in the pictures taken, whose visual weight is greater than a preset value, generates a partition independent image for each object partition, inputs each partition independent image into a graph-to-text model to generate a corresponding partition text description, the object text description contains boundary information of partition sub-elements in the corresponding object partition, the partition sub-elements are image elements constituting the partition independent image, the boundary information includes type information of the partition sub-elements on both sides of the boundary, and optimizes the picture quality of the corresponding boundary area of ​​the pictures taken based on the boundary information, which can effectively improve the optimization effect of the camera pictures taken quality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for optimizing the quality of pictures shot by a camera and a camera. Background Art

[0002] Cameras are one of the most widely used electronic devices in people's daily lives, playing an important role in many application scenarios such as security monitoring, mobile phone photography, video chat and remote conferencing. The quality of the images captured by the camera is affected by a variety of internal and external factors. For example, internal factors include the camera's resolution, photosensitive elements, and the imaging quality of the optical lens; external factors include the light intensity of the shooting environment, the distance between the subject and the camera, and the relative movement speed. More specifically, the quality of images captured by low-resolution cameras is usually not as good as that of high-resolution cameras, and the quality of images captured in dark and strong light environments will also be worse than that in environments with uniform light.

[0003] In the prior art, optimization methods such as noise reduction and image resolution enhancement are usually used to improve the quality of the captured images. For example, commonly used noise reduction models include DnCNN (Denoising Convolutional Neural Network), FFDNet (Fast and Flexible Denoising Network) and CBDNet (Convolutional Blind Denoising Network), which use different algorithms to reduce the noise in the image. Commonly used models for improving image resolution include SRCNN (Super-Resolution Convolutional Neural Network), ESPCN (Efficient Sub-Pixel Convolutional Neural Network), EDSR (Enhanced Deep Super-Resolution Network) and SRGAN (Super-Resolution Generative Adversarial Network) and other super-resolution models. Although these models can optimize the image quality to a certain extent, their effects often cannot meet the user's usage requirements, and in some cases, negative optimization phenomena such as reduced image quality may even occur. For example, insufficient border optimization may result in blurred image edges or excessive graininess, or excessive border optimization may result in overly sharp edges, making the image lose its realism. Summary of the invention

[0004] Based on the above problems, the present invention proposes a method for optimizing the quality of camera pictures taken and a camera, which can effectively improve the optimization effect of the quality of camera pictures taken.

[0005] In view of this, a first aspect of the present invention provides a method for optimizing the quality of a captured image of a camera, comprising:

[0006] Get the camera's captured image;

[0007] Performing object recognition on the captured image, wherein the object includes static objects and dynamic objects in the captured image;

[0008] Dividing the captured image into a plurality of object partitions according to the object recognition result, so that each object partition contains a key object, wherein the key object is an object in the captured image whose visual weight is greater than a preset value;

[0009] generating partition-independent images of each object partition;

[0010] Input each partitioned independent image into the image-to-text model to generate a corresponding partitioned text description, wherein the object text description includes boundary information of partitioned sub-elements in the corresponding object partition, wherein the partitioned sub-elements are image elements constituting the partitioned independent image, and the boundary information includes type information of the partitioned sub-elements on both sides of the boundary;

[0011] The image quality of the corresponding boundary area of ​​the captured image is optimized based on the boundary information.

[0012] Furthermore, the step of dividing the captured image into a plurality of object partitions according to the object recognition result specifically includes:

[0013] Constructing an object sequence of the captured image, wherein the object sequence is a nested sequence, in which a parent object and a child object are represented in a nested form;

[0014] Performing visual weight calculation on objects at each nesting level in the object sequence;

[0015] The object partitions are divided according to the visual weight of each object in the object sequence.

[0016] Furthermore, the step of performing visual weight calculation on objects at each nested level in the object sequence specifically includes:

[0017] Determine the image range corresponding to each object on the shooting screen;

[0018] Compute the size saliency of each object based on the image extent , color saliency , Position prominence and shape saliency ,in 1 to A positive integer between is the number of objects in the object sequence;

[0019] Calculate the visual weight of each object:

[0020] ,

[0021] in , and Size saliency , color saliency and positional salience The weight coefficient of .

[0022] Furthermore, the size saliency of each object is calculated based on the image range , color saliency , Position prominence and shape saliency The steps specifically include:

[0023] Get the The pixel area of ​​the image range of the object And the total pixel area of ​​the shooting picture ;

[0024] Calculate the Size saliency of objects:

[0025] ;

[0026] Get the The saturation of each pixel in the image range of objects and brightness ,in 1 to A positive integer between For the The number of pixels within the image range of the object, and , ;

[0027] Calculate the Color saliency of objects:

[0028] ;

[0029] Get the The pixel position of each pixel within the image range of the object And the central pixel position of the captured image ,in:

[0030] , is the rounding symbol;

[0031] Calculate the Position saliency of objects:

[0032] ;

[0033] Calculate the first The perimeter of the image range of the object , where the perimeter of the image range is represented by the number of pixels on the edge line of the image range;

[0034] Calculate the Shape saliency of objects:

[0035] .

[0036] Furthermore, the step of dividing the object partitions according to the visual weight of each object in the object sequence specifically includes:

[0037] Obtaining a preconfigured visual weight threshold, where the visual weight threshold is used to distinguish between key objects and non-key objects;

[0038] Constructing a candidate key object sequence according to the visual weight threshold, wherein the candidate key object sequence is a subset of the object sequence, and the candidate key objects in the candidate key object sequence are objects in the object sequence whose visual weight is greater than the visual weight threshold;

[0039] Traversing the determined candidate key objects in the candidate key object sequence to determine whether the candidate key object has a parent object in the candidate key object sequence;

[0040] When any candidate key object in the candidate key object sequence does not have a parent object in the candidate key object sequence, the corresponding candidate key object is determined as a key object.

[0041] Furthermore, after the step of determining the corresponding candidate key object as the key object, the method further includes:

[0042] Determining the number of key objects in the captured image;

[0043] Determining whether the number of key objects in the captured image is greater than 1;

[0044] When the number of key objects on the shooting picture is greater than 1, determining the boundary of each key object on the shooting picture, and the image area surrounded by the boundary of the key object is the image range of the key object;

[0045] Constructing an object midline on the shooting picture, wherein the object midline is a middle dividing line between two key objects on the shooting picture;

[0046] The captured image is divided into a plurality of object partitions based on the object centerline.

[0047] Furthermore, the step of constructing the object centerline on the shooting picture specifically includes:

[0048] Constructing a plurality of positioning auxiliary lines on the shooting picture, wherein the positioning auxiliary lines include horizontal positioning auxiliary lines and vertical positioning auxiliary lines, wherein the horizontal positioning auxiliary lines and the vertical positioning auxiliary lines extend from one end of the shooting picture to the other end of the shooting picture, and each positioning auxiliary line penetrates at least two key objects;

[0049] From each positioning auxiliary line, intercept a positioning auxiliary line segment that falls outside the image range of the key object and has boundary pixels of the key object as both end points;

[0050] Determine the midline pixel point of the midpoint of the positioning auxiliary line segment;

[0051] The midline pixels are connected to obtain the object midline.

[0052] Furthermore, the step of optimizing the image quality of the corresponding boundary area of ​​the captured image based on the boundary information specifically includes:

[0053] Matching the image range corresponding to the partition sub-elements on both sides of the boundary of the key object in each object partition according to the partition text description;

[0054] Segmenting the boundary of the key object into a plurality of boundary line segments according to the image range of the partition sub-element, each boundary line segment is associated with two partition sub-elements, and adjacent boundary line segments contain at least one different partition sub-element;

[0055] The image quality on both sides of the boundary segment is optimized according to the type of partition sub-element.

[0056] Furthermore, the step of optimizing the image quality on both sides of the boundary segment according to the type of the partition sub-element specifically includes:

[0057] Determine the types of the two partition sub-elements on both sides of each boundary segment;

[0058] Load the texture templates corresponding to the two types of partition sub-elements;

[0059] Acquire pre-configured boundary optimization parameters corresponding to two partition sub-elements, wherein the boundary optimization parameters include a first contrast distribution and a first saturation distribution along a direction parallel to the boundary, and a second contrast distribution and a second saturation distribution along a direction perpendicular to the boundary;

[0060] The images on both sides of the boundary segment are optimized based on the texture template and the boundary optimization parameters.

[0061] A second aspect of the present invention provides a camera, comprising a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the method for optimizing the quality of captured images as described in any one of the first aspects of the present invention.

[0062] The present invention proposes a method for optimizing the quality of camera pictures taken and a camera. The method performs object recognition on the camera pictures taken, divides the pictures taken into a plurality of object partitions according to the object recognition results, so that each object partition contains a key object, the key object is an object in the pictures taken, whose visual weight is greater than a preset value, generates a partition independent image for each object partition, inputs each partition independent image into a graph-to-text model to generate a corresponding partition text description, the object text description contains boundary information of partition sub-elements in the corresponding object partition, the partition sub-elements are image elements constituting the partition independent image, the boundary information includes type information of the partition sub-elements on both sides of the boundary, and optimizes the picture quality of the corresponding boundary area of ​​the pictures taken based on the boundary information, which can effectively improve the optimization effect of the camera pictures taken quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flow chart of a method for optimizing the quality of captured images of a camera provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0065] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0066] In the description of the present invention, the term "multiple" refers to two or more. Unless otherwise clearly defined, the orientation or positional relationship indicated by the terms "upper", "lower", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation of the present invention. The terms "connection", "installation", "fixation", etc. should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In addition, the terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, the meaning of "multiple" is two or more.

[0067] In the description of this specification, the description of the terms "one embodiment", "some implementations", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0068] A method for optimizing the quality of captured images of a camera and a camera provided according to some embodiments of the present invention will be described below with reference to the accompanying drawings.

[0069] like Figure 1 As shown, the first aspect of the present invention proposes a method for optimizing the quality of a captured image of a camera, comprising:

[0070] Get the camera's captured image;

[0071] Performing object recognition on the captured image, wherein the object includes static objects and dynamic objects in the captured image;

[0072] Dividing the captured image into a plurality of object partitions according to the object recognition result, so that each object partition contains a key object, wherein the key object is an object in the captured image whose visual weight is greater than a preset value;

[0073] generating partition-independent images of each object partition;

[0074] Input each partitioned independent image into the image-to-text model to generate a corresponding partitioned text description, wherein the object text description includes boundary information of partitioned sub-elements in the corresponding object partition, wherein the partitioned sub-elements are image elements constituting the partitioned independent image, and the boundary information includes type information of the partitioned sub-elements on both sides of the boundary;

[0075] The image quality of the corresponding boundary area of ​​the captured image is optimized based on the boundary information.

[0076] Specifically, the static object is a stationary object in the captured image, such as a building, a road or a plant, and the dynamic object is a moving object in the captured image, such as a human body, a car or an animal. In the captured image of the camera, in addition to the visual elements that constitute the object itself, the boundary between objects with a large visual weight is one of the main visual elements that affect the visual effect of the captured image. In the step of dividing the captured image into a plurality of object partitions according to the object recognition result, each object partition is made to contain and only contain one key object, so that the boundary information of the partition sub-elements in each object partition can be more targeted for identification.

[0077] Furthermore, the step of generating a partition-independent image for each object partition specifically includes:

[0078] Extracting image content of each object partition from the captured image;

[0079] The pixel values ​​of a preset color are filled around the image content of each object partition to construct it into a partition independent image of uniform size.

[0080] Preferably, the pixel value of the preset color is a black pixel value or a white pixel value.

[0081] Preferably, the uniform size is equal to the size of the shooting picture. For example, the size of the shooting picture is , that is, it contains pixels, vertically including When the captured image is divided into object partitions, then in the step of generating a partition-independent image for each object partition, The size is The partitioned independent images each contain only the image content of one object partition, and except for the image content of the object partition extracted from the captured image, the remaining pixels are filled with black or white.

[0082] The image-to-text model is a natural language model trained based on deep learning technologies such as CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network), and includes an image encoder for extracting image features and a text decoder for generating corresponding natural language descriptions, wherein the image encoder is usually a CNN model and the text decoder is usually an RNN model. In order to enable the image-to-text model to grasp the potential relationship between images and texts and accurately generate corresponding natural language descriptions based on the image content, the image-to-text model needs to be trained using a large number of sample data sets consisting of image-text pairs.

[0083] In the technical solution of the present invention, any continuous area in an image that has an approximate color value as a whole is called an image element, such as the sky, clouds, hair, clothes, etc., and there will be boundaries between adjacent image elements. When optimizing the image quality, especially improving the image resolution, the lack of boundary details of the original image will cause these boundaries to be blurred, with large pixel granularity or edge distortion. In the technical solution of the above-mentioned implementation, each partition sub-element is a single image element, and the type information of the partition sub-element in the boundary information is specifically the object type to which the partition sub-element belongs, such as the partition sub-element is the sky or the partition sub-element is hair, etc.

[0084] Furthermore, the step of dividing the captured image into a plurality of object partitions according to the object recognition result specifically includes:

[0085] Constructing an object sequence of the captured image, wherein the object sequence is a nested sequence, in which a parent object and a child object are represented in a nested form;

[0086] Performing visual weight calculation on objects at each nesting level in the object sequence;

[0087] The object partitions are divided according to the visual weight of each object in the object sequence.

[0088] In the step of performing object recognition on the captured image, all objects in the captured image are recognized regardless of their size to generate corresponding object names, so as to construct an object sequence through these object names, and determine the subordinate relationship between objects during the recognition process. Specifically, in the subordinate relationship between the parent object and the child object, the image range of the parent object contains the image range of the child object, that is, the image range of the child object is a part of the image range of the parent object. For example, if there is a building in the captured image, and the doors, windows, etc. of the building can be seen in the captured image, then the recognized name list includes the building, doors and windows, and the doors and windows are subordinate to the building, that is, in the object sequence, the building is the parent object of the doors and windows, and the doors and windows are the child objects of the building. In this example, an object sequence can be expressed as follows:

[0089] {

[0090] "Building": {"door", "window"},

[0091] "ground": {"grass","road","square"},

[0092] …

[0093] }

[0094] In some cases, multiple objects of the same type may appear in the same shooting picture. In the technical solutions of some embodiments of the present invention, when there are multiple objects of the same type in the shooting picture, these objects are distinguished by serial numbers, for example:

[0095] {

[0096] "building": {"door", "window"},

[0097] "ground": {"grass","road","square"},

[0098] "Pedestrian 1": {...},

[0099] "Pedestrian 2": {...},

[0100] …

[0101] }

[0102] Furthermore, the step of performing visual weight calculation on objects at each nested level in the object sequence specifically includes:

[0103] Determine the image range corresponding to each object on the shooting screen;

[0104] Compute the size saliency of each object based on the image extent , color saliency , Position prominence and shape saliency ,in 1 to A positive integer between is the number of objects in the object sequence;

[0105] Calculate the visual weight of each object:

[0106] ,

[0107] in , and Size saliency , color saliency and positional salience The weight coefficient of .

[0108] Specifically, the visual weight of each object on the captured image is related to factors such as its size, color, position, and shape on the captured image. In the technical solution of the above-mentioned implementation, the visual weight of the corresponding object in the captured image is obtained by calculating the size saliency, color saliency, position saliency, and shape saliency of each object. The size saliency, color saliency, and position saliency take values ​​between 0 and 1.

[0109] Furthermore, the size saliency of each object is calculated based on the image range , color saliency , Position prominence and shape saliency The steps specifically include:

[0110] Get the The pixel area of ​​the image range of the object And the total pixel area of ​​the shooting picture ;

[0111] Calculate the Size saliency of objects:

[0112] ;

[0113] Get the The saturation of each pixel in the image range of objects and brightness ,in 1 to A positive integer between For the The number of pixels within the image range of the object, and , ;

[0114] Calculate the Color saliency of objects:

[0115] ;

[0116] Get the The pixel position of each pixel within the image range of the object And the central pixel position of the captured image ,in:

[0117] , is the rounding symbol;

[0118] Calculate the Position saliency of objects:

[0119] ;

[0120] Calculate the first The perimeter of the image range of the object , where the perimeter of the image range is represented by the number of pixels on the edge line of the image range;

[0121] Calculate the Shape saliency of objects:

[0122] .

[0123] Specifically, the size significance indicates the proportion of the pixel area of ​​the image range of the object in the captured image. The larger the proportion of the image range of the object, the more significant it is in the captured image. In the technical solution of the above-mentioned implementation, the pixel area of ​​the image range is represented by the number of pixels in the image range, and the size of the captured image is For example, the total pixel area of ​​the shooting picture .

[0124] The color saliency is calculated based on the saturation and brightness of each pixel within the image range of the object. The higher the saturation and brightness, the more salient the object is.

[0125] Based on the composition principle, the closer the image element is to the center of the image, the easier it is to attract attention. Therefore, the position saliency is calculated based on the distance between the pixels within the object image range and the center position of the captured image.

[0126] In the technical solution of the above embodiment, the shape saliency To reflect the The irregularity of the shape of the image range of an object is the smallest when its shape is circular, and its shape saliency is The more complex the edge contour is, the greater the irregularity of its shape is, and the corresponding shape saliency is The larger the value.

[0127] Furthermore, the step of dividing the object partitions according to the visual weight of each object in the object sequence specifically includes:

[0128] Obtaining a preconfigured visual weight threshold, where the visual weight threshold is used to distinguish between key objects and non-key objects;

[0129] Constructing a candidate key object sequence according to the visual weight threshold, wherein the candidate key object sequence is a subset of the object sequence, and the candidate key objects in the candidate key object sequence are objects in the object sequence whose visual weight is greater than the visual weight threshold;

[0130] Traversing the determined candidate key objects in the candidate key object sequence to determine whether the candidate key object has a parent object in the candidate key object sequence;

[0131] When any candidate key object in the candidate key object sequence does not have a parent object in the candidate key object sequence, the corresponding candidate key object is determined as a key object.

[0132] In the technical solution of the above-mentioned implementation mode, when the visual weight of any object in the photographed image is greater than or equal to the visual weight threshold, the object may be a key object or a sub-object of a key object. Conversely, when the visual weight of any object in the photographed image is less than the visual weight threshold, the object is determined to be a non-key object.

[0133] Further, after the step of traversing the candidate key object sequence to determine the candidate key object to determine whether the candidate key object has a parent object in the candidate key object sequence, the method further includes:

[0134] When any candidate key object in the candidate key object sequence has a parent object in the candidate key object sequence, the corresponding candidate key object is determined as a non-key object.

[0135] Furthermore, after the step of determining the corresponding candidate key object as the key object, the method further includes:

[0136] Determining the number of key objects in the captured image;

[0137] Determining whether the number of key objects in the captured image is greater than 1;

[0138] When the number of key objects on the shooting picture is greater than 1, determining the boundary of each key object on the shooting picture, and the image area surrounded by the boundary of the key object is the image range of the key object;

[0139] Constructing an object midline on the shooting picture, wherein the object midline is a middle dividing line between two key objects on the shooting picture;

[0140] The captured image is divided into a plurality of object partitions based on the object centerline.

[0141] In the technical solution of the above-mentioned embodiment, the image range in the captured picture can be divided into the image range of the key object and the image range of the non-key object with the continuity of the boundary pixels of the key object as the boundary, and the object centerline is the middle dividing line between the key objects on the captured picture, which means that the object centerline is between the two key objects on the captured picture, that is, it falls within the image range of the non-key object on the captured picture. By adopting the technical solution of the above-mentioned embodiment, by constructing the object centerline on the captured picture, a part of the image range of the non-key object can be divided into the object partitions containing each key object, so that the partition sub-elements in the boundary information not only include the image elements within the image range of the key object, but also include the image elements within the image range of the non-key object in the captured picture.

[0142] Furthermore, after the step of determining whether the number of key objects on the captured image is greater than 1, the method further includes:

[0143] When the number of key objects on the photographic picture is equal to 1, the entire photographic picture is determined as an object partition.

[0144] Furthermore, the step of constructing the object centerline on the shooting picture specifically includes:

[0145] Constructing a plurality of positioning auxiliary lines on the shooting picture, wherein the positioning auxiliary lines include horizontal positioning auxiliary lines and vertical positioning auxiliary lines, wherein the horizontal positioning auxiliary lines and the vertical positioning auxiliary lines extend from one end of the shooting picture to the other end of the shooting picture, and each positioning auxiliary line penetrates at least two key objects;

[0146] From each positioning auxiliary line, intercept a positioning auxiliary line segment that falls outside the image range of the key object and has boundary pixels of the key object as both end points;

[0147] Determine the midline pixel point of the midpoint of the positioning auxiliary line segment;

[0148] The midline pixels are connected to obtain the object midline.

[0149] Specifically, the horizontal positioning auxiliary line extending from one end of the shooting screen to the other end of the shooting screen means that the horizontal positioning auxiliary line extends from the left edge of the shooting screen to the right edge of the shooting screen so as to penetrate the entire shooting screen. Similarly, the vertical positioning auxiliary line extending from one end of the shooting screen to the other end of the shooting screen means that the vertical positioning auxiliary line extends from the top edge of the shooting screen to the bottom edge of the shooting screen so as to penetrate the entire shooting screen.

[0150] Furthermore, in the step of intercepting a positioning auxiliary line segment that falls outside the image range of the key object and has boundary pixels of the key object as both end points from each positioning auxiliary line, both end points of the positioning auxiliary line segment are boundary pixels of the key object, and both end points of the positioning auxiliary line segment fall on the boundaries of different key objects. The midpoint of the positioning auxiliary line segment refers to the middle point of the two end points of the positioning auxiliary line segment, that is, when the coordinate values ​​of the two end points of the positioning auxiliary line segment in the captured image are respectively , When , the coordinate value in the shooting picture is The point is the midpoint of the positioning auxiliary line segment.

[0151] In the technical solution of the above-mentioned implementation mode, since each positioning auxiliary line penetrates at least two key objects, a centerline pixel point can be obtained from each positioning auxiliary line segment. In the step of connecting the centerline pixel points to obtain the centerline of the object, a smooth curve is used to connect these centerline pixel points to obtain the centerline of the object.

[0152] In the technical solutions of some embodiments of the present invention, after the step of connecting the midline pixel points using a smooth curve to obtain the midline of the object, the method further includes:

[0153] Determine the tangent directions of the endpoints of each object's midline;

[0154] The object centerline is extended to the edge of the shooting picture along the tangent direction.

[0155] In the technical solutions of some other embodiments of the present invention, after the step of connecting the midline pixel points using a smooth curve to obtain the midline of the object, the method further includes:

[0156] Determine the tangent directions of the endpoints of each object's midline;

[0157] Calculating a first angle between the tangent direction and the horizontal direction and a second angle between the tangent direction and the longitudinal direction;

[0158] When the first angle is smaller than the second angle, connecting the endpoints of the object centerline using a horizontal line segment to extend the object centerline to the edge of the captured image;

[0159] When the first angle is greater than the second angle, using a longitudinal line segment to connect the endpoints of the object centerline to extend the object centerline to the edge of the shooting screen;

[0160] When the first angle is equal to the second angle, the endpoints of the object centerline are connected using either a horizontal line segment or a vertical line segment to extend the object centerline to the edge of the shooting picture.

[0161] Furthermore, the step of optimizing the image quality of the corresponding boundary area of ​​the captured image based on the boundary information specifically includes:

[0162] Matching the image range corresponding to the partition sub-elements on both sides of the boundary of the key object in each object partition according to the partition text description;

[0163] Segmenting the boundary of the key object into a plurality of boundary line segments according to the image range of the partition sub-element, each boundary line segment is associated with two partition sub-elements, and adjacent boundary line segments contain at least one different partition sub-element;

[0164] The image quality on both sides of the boundary segment is optimized according to the type of partition sub-element.

[0165] Furthermore, the step of matching the image range corresponding to the partition sub-elements on both sides of the boundary of the key object in each object partition according to the partition text description specifically includes:

[0166] Extracting type information of partition sub-elements from the partition text description;

[0167] Matching the types of the partition sub-elements one by one in the object partition to obtain an image range corresponding to each partition sub-element;

[0168] Determine a partition sub-element whose image range covers the boundary of the key object as a target partition sub-element;

[0169] The image range that holds the target partition sub-element.

[0170] In the technical solution of the above implementation, the two partition sub-elements associated with each boundary line segment are located on both sides of the corresponding boundary line segment, that is, one of the partition sub-elements is located within the image range of the corresponding key object, and the other partition sub-element is located outside the image range of the corresponding key object. Adjacent boundary line segments refer to two boundary line segments with a common endpoint.

[0171] Furthermore, the step of segmenting the boundary of the key object into a plurality of boundary line segments according to the image range of the partition sub-element specifically includes:

[0172] Determine the boundary line segment where the target partition sub-element falls on the boundary of the key object ,in , is the number of partition sub-elements in the object partition whose adjacent lines are the boundaries of the key object;

[0173] Establish the adjacent relationship between the target partition sub-elements according to the image range of the partition sub-elements ,in , Respectively The partition sub-element and partition sub-elements, and ;

[0174] The boundary of the key object is divided into the intersection line segments of several boundary line segments of the target partition sub-elements with adjacent relationships:

[0175] .

[0176] For example, when the partition sub-element With partition sub-elements , If the boundary of the key object is the adjacent line, the partition sub-element With partition sub-elements The intersection line segment and partition sub-elements With partition sub-elements The intersection line segment are respectively determined as the boundary line segments.

[0177] Furthermore, the step of optimizing the image quality on both sides of the boundary segment according to the type of the partition sub-element specifically includes:

[0178] Determine the types of the two partition sub-elements on both sides of each boundary segment;

[0179] Load the texture templates corresponding to the two types of partition sub-elements;

[0180] Acquire pre-configured boundary optimization parameters corresponding to two partition sub-elements, wherein the boundary optimization parameters include a first contrast distribution and a first saturation distribution along a direction parallel to the boundary, and a second contrast distribution and a second saturation distribution along a direction perpendicular to the boundary;

[0181] The images on both sides of the boundary segment are optimized based on the texture template and the boundary optimization parameters.

[0182] Furthermore, the step of optimizing the images on both sides of the boundary segment based on the texture template and the boundary optimization parameters specifically includes:

[0183] Configuring a target resolution, wherein the target resolution is greater than the resolution of the captured image;

[0184] Based on the texture template, pixel scaling and texture filling are performed on the images on both sides of the boundary segment so that the resolution of the images on both sides of the boundary segment reaches the target resolution, and the contrast and saturation of the images on both sides of the boundary segment match the boundary optimization parameters.

[0185] Furthermore, the step of performing pixel scaling and texture filling on the images on both sides of the boundary segment based on the texture template specifically includes:

[0186] Using null-value pixels to perform interpolation to expand the resolution of the image on both sides of the boundary segment to the target resolution, so that each pixel point in the image before expansion is evenly distributed in the image after expansion, wherein the null-value pixels refer to pixels points to which no color value is assigned;

[0187] The color value of the null pixel is generated according to the texture in the texture template, so that the image textures on both sides of the boundary line segment match the texture in the texture template.

[0188] In some other implementations of the present invention, in expanding the resolution of the image on both sides of the boundary segment to the target resolution, monochrome pixels may also be used for interpolation.

[0189] Specifically, the first contrast distribution, the first saturation distribution, the second contrast distribution, and the second saturation distribution are preconfigured distribution curves. In the step of performing pixel scaling and texture filling on the images on both sides of the boundary line segment based on the texture template, the corresponding contrast distribution curve and saturation distribution curve are matched according to the length and width of the images on both sides of the boundary line segment, so that the contrast and saturation of the images on both sides of the boundary line segment along the direction parallel to the boundary match the distribution curves of the first contrast distribution and the first saturation distribution, and the contrast and saturation of the images on both sides of the boundary line segment along the direction perpendicular to the boundary match the distribution curves of the second contrast distribution and the second saturation distribution.

[0190] Furthermore, after the step of optimizing the image quality on both sides of the boundary segment according to the type of the partition sub-element, the method further includes:

[0191] Configure the image block size, the image block size includes the number of horizontal pixels and the number of vertical pixels, wherein the number of horizontal pixels is much smaller than , the number of vertical pixels is much smaller than ;

[0192] constructing other image regions in the object partition except the images on both sides of the boundary line segment into non-boundary independent images with a size of the image block;

[0193] Inputting the non-boundary independent image into a super-resolution reconstruction model for super-resolution reconstruction, wherein the super-resolution reconstruction model may be a SRCNN model, an ESPCN model, an EDSR model or a SRGAN model;

[0194] The images on both sides of the boundary line segment and the non-boundary independent image are spliced ​​to obtain an optimized shooting picture.

[0195] A second aspect of the present invention provides a camera, comprising a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the method for optimizing the quality of captured images as described in any one of the first aspects of the present invention.

[0196] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0197] According to the embodiments of the present invention as described above, these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made based on the above description. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can make good use of the present invention and the modified use based on the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for optimizing the quality of a camera shot image, characterized in that: include: Get the camera's captured image; Performing object recognition on the captured image, wherein the object includes static objects and dynamic objects in the captured image; Dividing the captured image into a plurality of object partitions according to the object recognition result, so that each object partition contains a key object, wherein the key object is an object in the captured image whose visual weight is greater than a preset value; generating partition-independent images of each object partition; Input each partition independent image into the image-to-text model to generate a corresponding partition text description, wherein the partition text description includes boundary information of partition sub-elements in the corresponding object partition, wherein the partition sub-elements are image elements constituting the partition independent image, and the boundary information includes type information of the partition sub-elements on both sides of the boundary; Optimizing the image quality of a corresponding boundary area of ​​the captured image based on the boundary information; The step of optimizing the image quality of the corresponding boundary area of ​​the captured image based on the boundary information specifically includes: Matching the image range corresponding to the partition sub-elements on both sides of the boundary of the key object in each object partition according to the partition text description; Segmenting the boundary of the key object into a plurality of boundary line segments according to the image range of the partition sub-element, each boundary line segment is associated with two partition sub-elements, and adjacent boundary line segments contain at least one different partition sub-element; The image quality on both sides of the boundary segment is optimized according to the type of the partition sub-element; The steps of optimizing the image quality on both sides of the boundary segment according to the type of the partition sub-element specifically include: Determine the types of the two partition sub-elements on both sides of each boundary segment; Load the texture templates corresponding to the two types of partition sub-elements; Acquire pre-configured boundary optimization parameters corresponding to two partition sub-elements, wherein the boundary optimization parameters include a first contrast distribution and a first saturation distribution along a direction parallel to the boundary, and a second contrast distribution and a second saturation distribution along a direction perpendicular to the boundary; Optimizing the images on both sides of the boundary segment based on the texture template and the boundary optimization parameters; The step of optimizing the images on both sides of the boundary segment based on the texture template and the boundary optimization parameters specifically includes: Configuring a target resolution, wherein the target resolution is greater than the resolution of the captured image; Based on the texture template, pixel scaling and texture filling are performed on the images on both sides of the boundary segment so that the resolution of the images on both sides of the boundary segment reaches the target resolution, and the contrast and saturation of the images on both sides of the boundary segment match the boundary optimization parameters.

2. The method for optimizing the quality of camera images according to claim 1, characterized in that: The step of dividing the captured image into a plurality of object partitions according to the object recognition result specifically includes: Constructing an object sequence of the captured image, wherein the object sequence is a nested sequence, in which a parent object and a child object are represented in a nested form; Performing visual weight calculation on objects at each nesting level in the object sequence; The object partitions are divided according to the visual weight of each object in the object sequence.

3. The method for optimizing the quality of camera images according to claim 2, characterized in that: The step of performing visual weight calculation on objects at each nested level in the object sequence specifically includes: Determine the image range corresponding to each object on the shooting screen; Calculate the size saliency SS of each object based on the image range i , Color Saliency CS i , Position Prominence PS i and shape salience OS i , where i is 1 to n obj A positive integer between obj is the number of objects in the object sequence; Calculate the visual weight of each object: W i =(w1·SS i +w2·CS i +w3·PS i )·(1+OS i ), Where w1, w2 and w3 are size significance SS i , Color Saliency CS i , Position Prominence PS i Weight coefficients, where w1>w2>w3.

4. The method for optimizing the quality of camera pictures according to claim 3, characterized in that: Calculate the size saliency SS of each object based on the image range i , Color Saliency CS i , Position Prominence PS i and shape salience OS i The steps specifically include: Get the pixel area A of the image range of the i-th object i and the total pixel area A0 of the captured image; Calculate the size saliency of the i-th object: Get the saturation SA of each pixel in the image range of the i-th object ij and brightness BR ij , where j is 1 to n_pixcel i A positive integer between n_pixcel i is the number of pixels within the image range of the i-th object, and SA ij ∈[0,1],BR ij ∈[0,1]; Calculate the color saliency of the i-th object: Get the pixel position (x) of each pixel within the image range of the i-th object ij ,y ij ) and the central pixel position (x0, y0) of the captured image, wherein: is the rounding symbol; Calculate the position saliency of the i-th object: Calculate the perimeter L of the image range of the i-th object i , where the perimeter of the image range is represented by the number of pixels on the edge line of the image range; Calculate the shape saliency of the i-th object:

5. The method for optimizing the quality of camera pictures according to claim 2, characterized in that: The step of dividing the object partitions according to the visual weight of each object in the object sequence specifically comprises: Obtaining a preconfigured visual weight threshold, where the visual weight threshold is used to distinguish between key objects and non-key objects; Constructing a candidate key object sequence according to the visual weight threshold, wherein the candidate key object sequence is a subset of the object sequence, and the candidate key objects in the candidate key object sequence are objects in the object sequence whose visual weight is greater than the visual weight threshold; Traversing the determined candidate key objects in the candidate key object sequence to determine whether the candidate key object has a parent object in the candidate key object sequence; When any candidate key object in the candidate key object sequence does not have a parent object in the candidate key object sequence, the corresponding candidate key object is determined as a key object.

6. The method for optimizing the quality of camera pictures according to claim 5, characterized in that: After the step of determining the corresponding candidate key object as the key object, the method further includes: Determining the number of key objects in the captured image; Determining whether the number of key objects in the captured image is greater than 1; When the number of key objects on the shooting picture is greater than 1, determining the boundary of each key object on the shooting picture, and the image area surrounded by the boundary of the key object is the image range of the key object; Constructing an object midline on the shooting picture, wherein the object midline is a middle dividing line between two key objects on the shooting picture; The captured image is divided into a plurality of object partitions based on the object centerline.

7. The method for optimizing the quality of camera pictures according to claim 6, characterized in that: The step of constructing the object centerline on the shooting picture specifically includes: Constructing a plurality of positioning auxiliary lines on the shooting picture, wherein the positioning auxiliary lines include horizontal positioning auxiliary lines and vertical positioning auxiliary lines, wherein the horizontal positioning auxiliary lines and the vertical positioning auxiliary lines extend from one end of the shooting picture to the other end of the shooting picture, and each positioning auxiliary line penetrates at least two key objects; Cutting off from each positioning auxiliary line a positioning auxiliary line segment that falls outside the image range of the key object and has boundary pixels of the key object as both end points; Determine the midline pixel point of the midpoint of the positioning auxiliary line segment; The midline pixels are connected to obtain the object midline.

8. A camera, characterized in that: It comprises a memory and a processor, wherein the processor executes a computer program stored in the memory to implement the method for optimizing the quality of captured images as described in any one of claims 1 to 7.

Images