An image processing method, apparatus, electronic device, and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium; the application can acquire a shooting image and size information of the shooting image; the shooting image is extracted according to the shooting image and the size information of the shooting image, and a plurality of sub-images are obtained; local features of all the sub-images are spliced; global features of the shooting image are selected to determine a region of target image features in the shooting image; the region of the target image features is taken as a focus point to obtain a modified shooting image. In the application, the target image features in the shooting image are identified, the region of the target image features is taken as the focus point, the shooting image is modified, and the modified shooting image is obtained. Therefore, the acquired panoramic image can be processed, and the problem that some important scenes or figures in the image are greatly distorted can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Panoramic images typically refer to photographs taken within the normal effective field of view of human eyes (approximately 90 degrees horizontally and 70 degrees vertically) or including peripheral vision (approximately 180 degrees horizontally and 90 degrees vertically), or even encompassing a complete 360-degree scene.

[0003] However, when shooting panoramic images, the resulting images often exhibit significant distortion of important objects or people, thus failing to meet the shooting requirements of photographers. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, and storage medium that can process acquired panoramic images to improve the problem of significant distortion in some important objects or people in the images.

[0005] On one hand, embodiments of this application provide an image processing method, including:

[0006] Acquire the captured image and its dimensions;

[0007] Based on the captured image and its size information, the size, sliding step, and sliding direction of the sliding window are determined. Based on the size, sliding step, and sliding direction of the sliding window, the captured image is extracted to obtain multiple sub-images. Among them, at least one overlapping area is included in two adjacent sub-images.

[0008] The sub-image is subjected to feature recognition processing to obtain local features, which are the image features of the sub-image;

[0009] All the local features are stitched together to obtain global features, which are the image features of the captured image.

[0010] Feature selection processing is performed on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image;

[0011] The region containing the target image features is used as the focus point to modify the captured image, resulting in a modified captured image.

[0012] On the other hand, embodiments of this application also provide an image processing apparatus, including:

[0013] The acquisition unit is used to acquire the captured image and its size information.

[0014] The extraction unit is used to determine the size, sliding step and sliding direction of the set sliding window according to the captured image and the size information of the captured image, and to extract the captured image according to the size, sliding step and sliding direction to obtain multiple sub-images, wherein at least one overlapping area is included in two adjacent sub-images.

[0015] The recognition unit is used to perform feature recognition processing on the sub-image to obtain local features, wherein the local features are the image features of the sub-image;

[0016] The stitching unit is used to stitch together the local features of all the sub-images to obtain the global features of the captured image;

[0017] The selection unit is used to perform feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image;

[0018] The modification unit is used to modify the captured image by taking the area where the target image features are located as the focus point, so as to obtain the modified captured image.

[0019] On the other hand, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the image processing methods provided in embodiments of this application.

[0020] On the other hand, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the image processing methods provided in embodiments of this application.

[0021] In this application, the image processing device can identify target image features in a captured image, and use the area where the target image features are located as the focus point to modify the captured image, thereby obtaining a modified captured image. This allows for the processing of acquired panoramic images, improving the problem of significant distortion in some important objects or people within the image. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a scene illustrating the image processing method provided in an embodiment of this application;

[0024] Figure 2 This is a schematic flowchart of the image processing method provided in the embodiments of this application;

[0025] Figure 3 This is a flowchart illustrating the method for determining the size, sliding step length, and sliding direction of a sliding window according to an embodiment of this application.

[0026] Figure 4 This is a flowchart illustrating a method for performing feature recognition processing on a sub-image to obtain local features, as provided in an embodiment of this application.

[0027] Figure 5 This is a flowchart illustrating the method for obtaining global features of a captured image provided in an embodiment of this application;

[0028] Figure 6 This is a flowchart illustrating the method for determining target image features in a captured image and the region where the target image features are located in the captured image, as provided in an embodiment of this application.

[0029] Figure 7 This is a flowchart illustrating the method for obtaining a modified next frame image provided in an embodiment of this application;

[0030] Figure 8 This is a schematic flowchart of an embodiment of the image processing method provided in this application;

[0031] Figure 9 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application;

[0032] Figure 10 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] This application provides an image processing method, apparatus, electronic device, and storage medium.

[0035] Specifically, the image processing device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0036] In some embodiments, the image processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the image processing method of this application being implemented by the multiple servers.

[0037] In some embodiments, the server may also be implemented as a terminal.

[0038] For example, refer to Figure 1 The electronic device can be a server, which integrates an image processing device. In this embodiment, the server is used to acquire captured images and their size information; determine the size, sliding step, and sliding direction of a sliding window based on the captured images and their size information; extract multiple sub-images from the captured images based on the size, sliding step, and sliding direction, wherein at least one overlapping region is included in two adjacent sub-images; perform feature recognition processing on the sub-images to acquire local features, which are the image features of the sub-images; stitch together the local features of all sub-images to obtain the global features of the captured image; perform feature selection processing on the global features of the captured image to determine the target image features and the region where the target image features are located in the captured image; use the region where the target image features are located as the focus point to modify the captured image to obtain a modified captured image.

[0039] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0040] In this embodiment, an image processing method is provided, such as... Figure 2 As shown, the specific process of this image processing method can be as follows:

[0041] S110: Acquire the captured image and its size information.

[0042] The captured image can be the current frame image obtained when shooting video, or it can be a pre-shot image; the captured image can be a panoramic image or a non-panoramic image. In some embodiments, the captured image can be a panoramic image that surrounds the viewer in the horizontal direction from 0 to 360° and in the vertical direction from 0 to 180°.

[0043] The size information of the captured image can include its length, width, and field of view.

[0044] Obtaining the captured image and its size information can be achieved either by directly acquiring the size information of the captured image during the shooting process using the shooting device, or by processing the captured image to obtain its length and width.

[0045] S120. Based on the captured image and its size information, determine the size, sliding step, and sliding direction of the sliding window. Based on the size, sliding step, and sliding direction of the sliding window, extract the captured image to obtain multiple sub-images. Among them, at least one overlapping area is included in two adjacent sub-images.

[0046] A sliding window, also known as a sliding frame window, is used to improve data accuracy by expanding the range of values ​​at a given point to include that point within a larger region. This region is then used for calculations; the region is the window. A sliding window can frame a time series data within a specified unit length to calculate statistical indicators. It's analogous to a slider of a specified length moving on a ruler; each unit movement provides feedback on the image data within the slider.

[0047] The size of the sliding window refers to the area within which the sliding window extracts images from the captured image. The sliding window can be of any shape. In some embodiments, the sliding window can be a rectangular frame, and its size can include the length and width of the rectangular frame. In some embodiments, the sliding window can be a square frame, and its size can include the side length of the square frame.

[0048] The sliding direction of the sliding window is the direction in which the sliding window slides when the captured image is slid, and the sliding direction of the sliding window can be arbitrary. In some embodiments, the sliding direction of the sliding window is along the length direction of the captured image.

[0049] The sliding step size of the sliding window is the distance the sliding window slides each time it slides while capturing an image. The sliding step size of the sliding window can be set according to the size information of the captured image. In some embodiments, when the sliding direction of the sliding window is along the length direction of the captured image, the sliding step size of the sliding window is equal to the length of the captured image divided by the number of times the sliding window slides.

[0050] In some embodiments, such as Figure 3As shown, the method for determining the size, sliding step, and sliding direction of the sliding window based on the size information of the captured image includes:

[0051] S121. Determine the field of view of the captured image in a preset direction, where the preset direction is the sliding direction of the sliding window.

[0052] In optical engineering, the field of view (FOV) is also called the field of view, and its size determines the field of view of an optical instrument. In the embodiments of this application, when the captured image is a panoramic image, the field of view of the panoramic image is 0 to 360° in the horizontal direction and 0 to 180° in the vertical direction.

[0053] The preset direction refers to a direction preset by a person. In some embodiments, the preset direction is horizontal, and the sliding window moves along the horizontal direction of the captured image when it moves.

[0054] S122. Based on the field of view of the captured image in a preset direction, select multiple reference points on the captured image, wherein the field of view between two adjacent reference points is the same, and the distance between the reference points is equal to the sliding step of the sliding window.

[0055] A reference point is a point on the captured image that serves as a reference for the sliding window's movement during sliding. In some embodiments, the reference point corresponds to a base position on the sliding window, making the sliding window's movement more accurate. In some embodiments, the reference point is set along a preset direction. In some embodiments, when the preset direction is horizontal, the position of the reference point in the vertical direction can be arbitrarily set. When the sliding window moves, the base position is used to align with the reference point, which can be a point or a line.

[0056] Having the same field of view between two adjacent reference points means that the field of view between two adjacent reference points is the same along a preset direction, or it can mean that the distance between two adjacent reference points is the same along a preset direction.

[0057] S123. Based on the position of the reference point on the captured image, determine the extraction range of the sliding window in the captured image. The length of the extraction range in the preset direction is greater than the distance between the reference points in the preset direction.

[0058] The extraction range refers to the area covered by the sliding window on the captured image. The image extracted based on the covered range is used as a sub-image.

[0059] The length of the extraction range in the preset direction is greater than the distance between the reference points in the preset direction. This means that there is an overlapping part between the positions of the sliding window after the two reference points are moved. In other words, it is used to characterize that at least one overlapping area is included in two adjacent sub-images.

[0060] Extracting images from a sliding window involves extracting the image within the window as it moves along a preset direction, each time the window slides across a certain number of steps. This extracted image is then treated as a separate image, denoted as a sub-image. When the window slides across n steps, n sub-images are obtained.

[0061] The overlapping region in two adjacent sub-images means that both adjacent sub-images contain the same image. In some embodiments, when the captured image is a panoramic image that surrounds the image in a 360° horizontal direction and a 180° vertical direction, the sliding window extracts the captured image along the horizontal direction. Each extracted sub-image includes overlapping regions in its front and rear portions that overlap with other adjacent sub-images.

[0062] S130. Perform feature recognition processing on the sub-image to obtain local features, which are the image features of the sub-image.

[0063] Feature recognition processing can refer to pixel-level image classification of sub-images, that is, labeling the object category to which each pixel in the image belongs. Object categories can include landscapes, buildings, and people. In the embodiments of this application, through feature recognition processing, landscapes, buildings, and people in the sub-image can be identified, and by segmenting the landscapes, buildings, and people in the sub-image, a mask map covering different image features can be determined. Here, the mask map refers to presenting the background and foreground (the foreground can include buildings and people) of the sub-image through images with different grayscale levels. Local features can refer to global features present in the sub-image, or they can refer to partial image features in the sub-image.

[0064] In some embodiments, such as Figure 4 As shown, methods for performing feature recognition processing on sub-images to obtain local features include:

[0065] S131. Determine the pixel category in the sub-image.

[0066] Pixel category, or pixel type, refers to the type of pixel. Different images have different pixel types, and the values ​​passed to the template parameters also differ for different pixel types. For example, pixel data types include CV_32U, CV_32S, CV_32F, CV_8U, CV_8UC3, etc.

[0067] Determining the pixel category in a sub-image means determining the pixel category of each pixel in the sub-image.

[0068] S132. Based on the pixel categories in the sub-image, segment the sub-image to obtain the sub-image mask.

[0069] A mask is an image that uses pixel intensity to reflect the strength of features within a sub-image. For example, in some implementations, background areas can be represented by black with lower grayscale values, while foreground areas can be represented by white or gray with higher grayscale values. A mask can effectively distinguish between foreground and background areas within a sub-image.

[0070] Sub-image segmentation, based on pixel categories within the sub-image, involves determining the category of each pixel (e.g., background, person, or vehicle) to divide the image into regions. Finally, the sub-image is transformed using a mask image. For example, in some embodiments, background regions can be represented by black with lower grayscale values, while regions belonging to people or vehicles can be represented by white or gray with higher grayscale values, resulting in a mask image.

[0071] S133. Determine the local features in the sub-image based on the mask image of the sub-image.

[0072] Determining local features in a sub-image can refer to determining background features, human features, or object features within the sub-image.

[0073] S140. All local features are stitched together to obtain global features, which are the image features of the captured image.

[0074] Since a captured image is composed of sub-images, it is understandable that the local features of a sub-image can constitute the global features of the captured image.

[0075] Local feature stitching of sub-images refers to stitching all sub-images together according to the mapping relationship between the sub-images and the captured image to obtain a mask image of the captured image. The mask image of the captured image includes the local features of each sub-image, and the local features of all sub-images constitute the global features of the captured image. In some embodiments, the captured image obtained after stitching includes a foreground portion and a background portion. The foreground portion may include people, buildings, or objects in the captured image, and the background portion may include scenery in the captured image. The grayscale values ​​of the foreground and background portions are different.

[0076] In some embodiments, such as Figure 5 As shown, methods for stitching together the local features of all sub-images to obtain the global features of the captured image include:

[0077] S141. Determine the positional relationship between the sub-image and the captured image.

[0078] Positional relationship refers to the position of a sub-image in the captured image. This positional relationship can be determined by the positional relationship between the sliding window and the reference point, or by the coordinate position of the sub-image in the captured image.

[0079] S142. Based on the positional relationship between the sub-image and the captured image, the local features of the sub-image are mapped to the captured image to obtain the global features of the captured image.

[0080] Global features refer to the process of superimposing the mask images of multiple sub-images onto the captured image according to their positional relationship, thereby obtaining a captured image with all local features, where all local features are the global features.

[0081] In order to avoid the problem of feature recognition duplication, when the overlapping areas between adjacent first and second sub-images share common first and second local features, the image processing method further includes:

[0082] Calculate the first feature strength of the first local feature and the second feature strength of the second local feature;

[0083] Compare the intensity of the first feature and the intensity of the second feature:

[0084] When the intensity of the first feature is greater than the intensity of the second feature, the first local feature is selected as the target feature within the overlapping region.

[0085] When the intensity of the first feature is less than that of the second feature, the second local feature is selected as the target feature within the overlapping region.

[0086] When the intensity of the first feature is equal to the intensity of the second feature, either the first local feature or the second local feature is selected as the target feature within the overlapping region.

[0087] S150. Perform feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image.

[0088] Feature selection processing refers to selecting features with high saliency from global features. Features with high saliency are those whose region has high pixel intensity, and high pixel intensity means that the gray value of the pixel is greater than a preset pixel value.

[0089] By comparing the pixel values ​​with preset pixel values, features with high saliency, i.e., features with high pixel intensity, can be identified. Therefore, in some embodiments, the pixel values ​​of global features can be compared with preset pixel thresholds, and image features with pixel values ​​greater than the pixel thresholds can be used as target image features.

[0090] In some embodiments, such as Figure 6 As shown, the method for performing feature selection processing on the global features of the captured image to determine the target image features and the regions where the target image features are located in the captured image includes:

[0091] S151. Perform pixel region division processing on the global features of the captured image to obtain multiple pixel regions.

[0092] A pixel region can refer to the location of each pixel in the global feature.

[0093] Since the mask image performs masking processing on local features in step S140, meaning the grayscale values ​​of pixels in the masked portion are set to zero, to reduce computational load, it is unnecessary to calculate the location of the masked pixels; only the location of the unmasked pixels needs to be calculated. Therefore, in some embodiments, since a mask image is used to divide the global features of the captured image into pixel regions, it is only necessary to confirm the location of pixels in the portion not masked by the mask image.

[0094] S152. Determine the pixel values ​​of the pixel region.

[0095] Determining the pixel value of a pixel region can refer to determining the grayscale value of each pixel.

[0096] S153. Compare the pixel values ​​of the pixel region with the preset pixel threshold to determine the first pixel region with a pixel value greater than the pixel threshold and the second pixel region with a pixel value less than the pixel threshold. The image features at the first pixel region are denoted as the target image features.

[0097] The preset pixel threshold can be set manually.

[0098] Comparison refers to determining the magnitude of pixel values ​​(grayscale). The first pixel region with a value greater than a pixel threshold is designated as a salient region, representing a feature of the target image. The second pixel region has a value less than the pixel threshold. In some embodiments, because the foreground portion of the captured image has a larger grayscale value and therefore a higher pixel intensity, the foreground portion with a value greater than the pixel threshold is designated as a salient region in the captured image.

[0099] S154. Based on the comparison results, the pixel values ​​of the first pixel region and the second pixel region are binarized to obtain the captured image after binarization.

[0100] Binarization refers to adjusting the pixel values ​​of the first pixel region with pixel values ​​greater than the pixel threshold to the maximum, and adjusting the pixel values ​​of the second pixel region with pixel values ​​less than the pixel threshold to the minimum, and then obtaining the mask image of the captured image after adjustment, which facilitates the differentiation of target image features.

[0101] S155. Based on the captured image after binarization, determine the region where the target image features are located in the captured image.

[0102] Based on the difference between the pixel values ​​of the first pixel region and the second pixel region in the binarized image, the first pixel region where the target image feature is located is determined, and the first pixel region is recorded as the region where the target image feature is located in the image.

[0103] S160. Use the area where the target image features are located as the focus point to modify the captured image and obtain the modified captured image.

[0104] Focusing refers to the process of adjusting the distance between a lens and the imaging plane during image acquisition to achieve a sharp image of the subject. The focus point refers to the object being photographed, which can be a single pixel or a region of pixels.

[0105] Using the region containing the target image features as the focus point means using the region containing the target features as the focus point so that the region containing the target image features is the area with the clearest image.

[0106] Modifying an image based on the focus point refers to modifying the target image features in the captured image as the most important target to be presented, so that the target image features in the modified captured image are the clearest and have the least distortion.

[0107] When shooting video, such as Figure 7 As shown, the method also includes:

[0108] S170, Obtain the focus point of the modified captured image.

[0109] Obtaining the focus point of the modified image refers to determining the target feature region in the captured image.

[0110] S180. Based on the focus point of the modified captured image, modify the next frame captured image to obtain the modified next frame captured image.

[0111] Modifying the next frame image means taking a picture of the next frame image with the target feature area of ​​the modified image as the focus point, and then modifying the next frame image with the target feature area of ​​the modified image as the clearest and least distorted position.

[0112] In some embodiments, the captured image in the modified captured image can be the current frame image in the video capture, and the next frame image can be the next frame image in the video capture.

[0113] The image processing method in this embodiment of the invention will be described below with reference to a specific application scenario.

[0114] Please see Figure 8This is a schematic diagram illustrating an example of the image processing method applied in an experimental scenario according to an embodiment of the present invention. The image processing method is applied to a server and includes:

[0115] S201. Obtain the captured image and its size information.

[0116] The captured image is a panoramic image, with a length of 1000mm and a width of 500mm.

[0117] The horizontal field of view of panoramic images is 0–360°, and the vertical field of view is 0–180°.

[0118] S202. Extract sub-images from the captured images.

[0119] Methods for extracting sub-images from panoramic images include cube projection, multi-view projection, and sliding window extraction.

[0120] Based on the captured images and their dimensions, four reference points were identified with coordinates (0, 500), (250, 500), (500, 500), and (750, 500). The sliding window had a step size of 250, a horizontal field of view of 180°, a radius of 1, and a horizontal sliding direction. Four sub-images were extracted using the sliding window.

[0121] Since the horizontal interval between the reference points of the four sub-images is converted to a field of view of 90°, and the horizontal distance of each sub-image is converted to a field of view of 180°, there is an overlapping area between adjacent sub-images.

[0122] S203. Perform feature recognition processing on the sub-image to obtain local features, which are the image features of the sub-image.

[0123] U 2 The net performs feature recognition processing on the sub-image. Specifically, the sub-image is input into the encoder-decoder network structure model to obtain 6 mask images with the same size as the input sub-image. The intensity of the 6 mask images is averaged and output to obtain the sub-image mask.

[0124] S204. The local features of all sub-images are stitched together to obtain the global features of the captured image.

[0125] Based on the positional relationship between the sub-image and the captured image, the mask image of the sub-image is mapped onto the captured image. When overlapping features appear in the overlapping area of ​​the sub-image, and the overlapping features are characterized as first local features and second local features that overlap with each other, the intensity of the overlapping features is compared, and the first local feature or the second local feature with greater intensity is selected to obtain the mask image of the captured image.

[0126] S205. Perform feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image.

[0127] The mask image of the captured image is binarized. Pixels with intensity values ​​greater than a preset threshold are adjusted to 255, while pixels with intensity values ​​less than the preset threshold are adjusted to 0. This yields a binary image of the captured image, facilitating the identification and selection of target image features and determining the region where the target image features are located within the captured image.

[0128] S206. Using the region where the target image features are located as the focus point, modify the captured image to obtain the modified captured image.

[0129] In this embodiment, by identifying image features in the panoramic image, high-intensity target image features and their salient regions in the captured image are determined. These salient regions are then used as focus points to modify the captured image or subsequent video images, resulting in a modified image. This allows for the processing of the acquired panoramic image, improving the problem of significant distortion in important objects or people within the image.

[0130] To better implement the above methods, this application also provides an image processing apparatus, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, laptop computer, personal computer, or similar device; the server can be a single server or a server cluster composed of multiple servers.

[0131] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the image processing device specifically integrated into the server as an example.

[0132] For example, such as Figure 9 As shown, the image processing apparatus may include:

[0133] The acquisition unit 301 is used to acquire the captured image and the size information of the captured image;

[0134] The extraction unit 302 is used to determine the size, sliding step and sliding direction of the set sliding window according to the captured image and the size information of the captured image, and extract the captured image according to the size, sliding step and sliding direction to obtain multiple sub-images, wherein at least one overlapping area is included in two adjacent sub-images.

[0135] The recognition unit 303 is used to perform feature recognition processing on the sub-image to obtain local features, which are the image features of the sub-image.

[0136] The stitching unit 304 is used to stitch together the local features of all sub-images to obtain the global features of the captured image;

[0137] Selection unit 305 is used to perform feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image.

[0138] Modification unit 306 is used to modify the captured image by taking the area where the target image features are located as the focus point, so as to obtain the modified captured image.

[0139] In some embodiments of this application, the extraction unit 302 is specifically used for:

[0140] Determine the field of view of the captured image in a preset direction, where the preset direction is the sliding direction of the sliding window;

[0141] Based on the field of view of the captured image in a preset direction, multiple reference points are selected on the captured image. The field of view of two adjacent reference points is the same, and the distance between the reference points is equal to the sliding step size of the sliding window.

[0142] Based on the position of the reference point on the captured image, the extraction range of the sliding window in the captured image is determined. The length of the extraction range in the preset direction is greater than the distance between the reference points in the preset direction.

[0143] In some embodiments of this application, the identification unit 303 is specifically used for:

[0144] Determine the pixel category in the sub-image;

[0145] Based on the pixel categories in the sub-image, the sub-image is segmented to obtain the sub-image mask image;

[0146] Based on the mask image of the sub-image, determine the local features in the sub-image.

[0147] In some embodiments of this application, the splicing unit 304 is specifically used for:

[0148] Determine the positional relationship between the sub-image and the captured image;

[0149] Based on the positional relationship between the sub-image and the captured image, the local features of the sub-image are mapped to the captured image to obtain the global features of the captured image.

[0150] In some embodiments of this application, the identification unit 303 is specifically used for:

[0151] The sub-image includes a first sub-image and a second sub-image that are adjacent in position. The overlapping region of the first sub-image includes a first local feature, and the overlapping region of the second sub-image includes a second local feature.

[0152] When the first local feature overlaps with the second local feature, the method includes:

[0153] Calculate the first feature strength of the first local feature and the second feature strength of the second local feature;

[0154] Compare the intensity of the first feature and the intensity of the second feature:

[0155] When the intensity of the first feature is greater than the intensity of the second feature, the first local feature is selected as the target feature within the overlapping region.

[0156] When the intensity of the first feature is less than that of the second feature, the second local feature is selected as the target feature within the overlapping region.

[0157] When the intensity of the first feature equals the intensity of the second feature, either the first local feature or the second local feature is selected as the target feature within the overlapping region.

[0158] In some embodiments of this application, the selection unit 305 is specifically used for:

[0159] The global features of the captured image are divided into pixel regions to obtain multiple pixel regions;

[0160] Determine the pixel values ​​of the pixel region;

[0161] The pixel values ​​of the pixel region are compared with a preset pixel threshold to determine the first pixel region with a pixel value greater than the pixel threshold and the second pixel region with a pixel value less than the pixel threshold. The image features at the first pixel region are recorded as the target image features.

[0162] Based on the comparison results, the pixel values ​​of the first pixel region and the second pixel region are binarized to obtain the captured image after binarization.

[0163] Based on the binarized captured image, determine the region where the target image features are located in the captured image.

[0164] In some embodiments of this application, the modification unit 306 is specifically used for:

[0165] Obtain the focus point of the modified captured image;

[0166] Based on the focus point of the modified captured image, the next captured image is modified to obtain the modified next captured image.

[0167] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0168] As described above, the image processing apparatus of this embodiment acquires the captured image and its size information by the acquisition unit 301; the extraction unit 302 determines the size, sliding step, and sliding direction of the set sliding window based on the captured image and its size information, and extracts the captured image according to the size, sliding step, and sliding direction to obtain multiple sub-images, wherein at least one overlapping region is included in two adjacent sub-images; the recognition unit 303 performs feature recognition processing on the sub-images to obtain local features, which are the image features of the sub-images; the stitching unit 304 stitches the local features of all sub-images to obtain the global features of the captured image; the selection unit 305 performs feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image; the modification unit 306 uses the region where the target image features are located as the focus point to modify the captured image to obtain the modified captured image. Therefore, this embodiment can process the acquired panoramic image to improve the problem of large distortion in some important objects or people in the image.

[0169] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0170] In some embodiments, the image processing apparatus may also be integrated into multiple electronic devices, such as multiple servers, with the image processing method of this application being implemented by the multiple servers.

[0171] In this embodiment, the electronic device will be described in detail as an image processing device, for example, such as... Figure 10 As shown, it illustrates a structural schematic diagram of the image processing apparatus involved in the embodiments of this application. Specifically:

[0172] The image processing may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 10 The structure of the image processing apparatus shown does not constitute a limitation on the image processing apparatus. It may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0173] Processor 401 is the control center of the image processing, connecting various parts of the image processing system via various interfaces and lines. It executes various functions of the SSS and processes data by running or executing software programs and / or modules stored in memory 402, and by calling data stored in memory 402. In some embodiments, processor 401 may include one or more processing cores; in some embodiments, processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 401.

[0174] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the image processing device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0175] The image processing apparatus also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0176] The image processing may also include an input module 404, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0177] The image processing device may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, allowing image processing to be performed via short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0178] The image processing apparatus may also include a sensor module, and in some embodiments the sensor module may include an image sensor that can acquire images.

[0179] Although not shown, image processing may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the image processing loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions.

[0180] In some embodiments, a computer program product is also provided, comprising a computer program or instructions that, when executed by a processor, implement the steps in any of the above-described image processing methods.

[0181] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0182] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0183] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods provided in embodiments of this application.

[0184] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0185] Since the instructions stored in the storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0186] The foregoing has provided a detailed description of an image processing method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, include: Acquire the captured image and its dimensions; The captured image is a panoramic image; Based on the captured image and its size information, the size, sliding step, and sliding direction of the sliding window are determined. Based on the size, sliding step, and sliding direction of the sliding window, the captured image is extracted to obtain multiple sub-images. Among them, at least one overlapping area is included in two adjacent sub-images. The sub-image is subjected to feature recognition processing to obtain local features, which are the image features of the sub-image; All the local features are stitched together to obtain global features, which are the image features of the captured image. Feature selection processing is performed on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image; The region containing the target image features is used as the focus point to modify the captured image, resulting in a modified captured image.

2. The image processing method according to claim 1, characterized in that, After obtaining the modified captured image, the method includes: Obtain the focus point of the modified captured image; Based on the focus point of the modified captured image, the next frame captured image is modified to obtain the modified next frame captured image.

3. The image processing method according to claim 1, characterized in that, The method for determining the size, sliding step, and sliding direction of the sliding window based on the captured image and its size information includes: Determine the field of view of the captured image in a preset direction, wherein the preset direction is the sliding direction of the sliding window; Based on the field of view of the captured image in a preset direction, multiple reference points are selected on the captured image, wherein the field of view between two adjacent reference points is the same, and the distance between the reference points is equal to the sliding step size of the sliding window. Based on the position of the reference point on the captured image, the extraction range of the sliding window in the captured image is determined, and the length of the extraction range in the preset direction is greater than the distance between the reference points in the preset direction.

4. The image processing method according to claim 1, characterized in that, The method for performing feature recognition processing on the sub-image to obtain local features includes: Determine the pixel category in the sub-image; Based on the pixel categories in the sub-image, the sub-image is segmented to obtain a mask image of the sub-image; Based on the mask image of the sub-image, local features in the sub-image are determined.

5. The image processing method according to claim 1, characterized in that, The method for stitching together the local features of all the sub-images to obtain the global features of the captured image includes: Determine the positional relationship between the sub-image and the captured image; Based on the positional relationship between the sub-image and the captured image, the local features of the sub-image are mapped to the captured image to obtain the global features of the captured image.

6. The image processing method according to claim 5, characterized in that, The sub-image includes a first sub-image and a second sub-image that are adjacent in position. The overlapping region of the first sub-image includes a first local feature, and the overlapping region of the second sub-image includes a second local feature. When the first local feature overlaps with the second local feature, the method includes: Calculate the first feature intensity of the first local feature and the second feature intensity of the second local feature; Compare the first feature intensity and the second feature intensity: When the intensity of the first feature is greater than the intensity of the second feature, the first local feature is selected as the target feature within the overlapping region; When the intensity of the first feature is less than the intensity of the second feature, the second local feature is selected as the target feature within the overlapping region; When the intensity of the first feature is equal to the intensity of the second feature, the first local feature or the second local feature is selected as the target feature within the overlapping region.

7. The image processing method according to claim 1, characterized in that, The method for performing feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image includes: The global features of the captured image are divided into pixel regions to obtain multiple pixel regions; Determine the pixel values ​​of the pixel region; The pixel values ​​of the pixel region are compared with a preset pixel threshold to determine a first pixel region with a pixel value greater than the pixel threshold and a second pixel region with a pixel value less than the pixel threshold. The image features at the first pixel region are recorded as target image features. Based on the comparison results, the pixel values ​​of the first pixel region and the pixel values ​​of the second pixel region are binarized to obtain the captured image after binarization. Based on the binarized captured image, determine the region in the captured image where the target image feature is located.

8. An image processing apparatus, characterized in that, include: The acquisition unit is used to acquire the captured image and its size information. The captured image is a panoramic image; The extraction unit is used to determine the size, sliding step and sliding direction of the set sliding window according to the captured image and the size information of the captured image, and to extract the captured image according to the size, sliding step and sliding direction to obtain multiple sub-images, wherein at least one overlapping area is included in two adjacent sub-images. The recognition unit is used to perform feature recognition processing on the sub-image to obtain local features, wherein the local features are the image features of the sub-image; The stitching unit is used to stitch together the local features of all the sub-images to obtain the global features of the captured image; The selection unit is used to perform feature selection processing on the global features of the captured image to determine the target image features in the captured image and the region where the target image features are located in the captured image; The modification unit is used to modify the captured image by taking the area where the target image features are located as the focus point, thereby obtaining the modified captured image.

9. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps of the image processing method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image processing method according to any one of claims 1 to 7.

Images