Image processing method and device, electronic equipment and storage medium
By identifying the central area of the mesh-carrying image in cryo-electron microscopy and expanding it to the full picture, the adaptability and speed problems of identifying mesh-carrying holes in the prior art are solved, and more efficient and accurate target area position recognition is achieved.
Patent Information
- Application Number
- CN202310921101.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-25
- Publication Date
- 2025-02-07
AI Technical Summary
In cryo-electron microscopy imaging, the prior art method of identifying online holes has problems such as poor adaptability and slow recognition speed.
By determining the central area of the network-loading image and gradually determining the location of all target area in the network-loading image based on the target area position in the central area. The method includes obtaining the image to be processed, extracting the effective area, converting it to grayscale format and thresholding, identifying the outline, intercepting the candidate area, performing Gaussian filtering and multi-thresholding processing to mark the position of the coarse-grained target area, and finally passing the grid matching algorithm Determine all target area locations.
It improves the efficiency and robustness of the image processing process, reduces the time to identify the location of the target area, and can correct the errors caused by the identification of individual target areas, making the results more accurate.
Smart Images
Figure CN119399091A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art
[0002] Cryo-electron microscopy (Cryo-EM) is a technique used for high-resolution imaging of biomacromolecules. In cryo-EM experiments, samples are usually placed on a thin carbon film and then quickly frozen at extremely low temperatures to preserve their original structure. Afterwards, the sample is sliced into thin slices and imaged by electron microscopy to obtain high-resolution 3D reconstruction images. Grids play an important role in cryo-EM imaging. They are thin metal grids used to support sample slices. The grid consists of dozens of evenly arranged square areas (called "squares"), and each square is composed of dozens of evenly arranged circular areas (called "holes"), which carry the sample to be observed. During the electron microscopy imaging process, it is necessary to find the positions of these holes in each square of each grid so that the electron beam can be irradiated onto the sample. Therefore, automatically identifying the holes on the grid is an important task in cryo-EM imaging. The related technology of identifying holes on the grid has the defects of poor adaptability and slow recognition speed. Summary of the invention
[0003] In view of this, the present disclosure proposes an image processing method, device, electronic device and storage medium, aiming to improve the efficiency and robustness of the image processing process.
[0004] According to a first aspect of the present disclosure, there is provided an image processing method, the method comprising:
[0005] Determining a grid image including a plurality of target areas, each of the target areas having the same shape and size and arranged in a grid format;
[0006] Determining a central area of the grid image;
[0007] Identify a target area position included in the central area;
[0008] The positions of all target regions in the grid image are determined based on the positions of the target regions in the central region.
[0009] In a possible implementation manner, determining a grid image including a plurality of target areas includes:
[0010] Get the image to be processed;
[0011] The effective area in the image to be processed is extracted to obtain a grid image including a plurality of target areas.
[0012] In a possible implementation, extracting the effective area in the image to be processed to obtain a grid image including a plurality of target areas includes:
[0013] Convert the image to be processed into a grayscale image and perform thresholding;
[0014] Performing contour recognition on the image to be processed after format conversion and thresholding, obtaining a binary mask area surrounded by the largest contour as a valid area;
[0015] The effective area is extracted from the image to be processed to obtain a grid image including a plurality of target areas.
[0016] In a possible implementation manner, determining the central area of the grid image includes:
[0017] intercepting a candidate region from the grid image;
[0018] The position of the coarse-grained target area in the candidate area is marked by image processing to obtain the central area of the grid image.
[0019] In a possible implementation manner, the extracting a candidate area from the grid image includes:
[0020] Determine the coordinates of the upper left corner of the grid image, and the width and height of the grid image;
[0021] The upper left corner coordinates of the candidate area and the width and height of the candidate area are calculated according to the preset scaling size, the width, height and upper left corner coordinates of the grid image, and the candidate area is cut out to obtain.
[0022] In a possible implementation, the step of marking the position of the coarse-grained target region in the candidate region by image processing to obtain the central region of the grid image includes:
[0023] Performing Gaussian filtering on the candidate area;
[0024] The candidate area after Gaussian filtering is subjected to multi-threshold processing, and a target threshold is determined from multiple thresholds to mask the candidate area to obtain a central area with a coarse-grained target area position marked.
[0025] In a possible implementation manner, the identifying the location of the target area included in the central area includes:
[0026] Extracting contour information in the central area and obtaining the area wrapped by each contour;
[0027] Perform contour screening according to the area wrapped by each contour to obtain at least one candidate contour;
[0028] Determine the position information of each candidate contour;
[0029] Shape screening is performed according to the position information of each candidate contour to obtain a target contour for marking the position of the target area.
[0030] In a possible implementation manner, the candidate contour is a circular contour, and determining the position information of each candidate contour includes:
[0031] Obtaining the contour point coordinates of the highest point, the lowest point, the leftmost point and the rightmost point of each candidate contour, and calculating the center point coordinates of the candidate contour and the radius value of the candidate contour according to the four contour point coordinates;
[0032] The corresponding position information is determined according to the contour point coordinates, the center point coordinates and the radius value of each candidate contour.
[0033] In a possible implementation, performing shape screening according to the position information of each candidate contour to obtain a target contour for marking the position of the target area includes:
[0034] Perform shape screening by performing at least one of boundary detection, circle detection and standard deviation detection based on the position information of each candidate contour, and delete the candidate contours that do not meet at least one detection requirement;
[0035] The candidate contour that passes the shape screening is determined as the target contour that marks the position of the target area.
[0036] In a possible implementation, the boundary detection is used to filter candidate contours whose coordinates of at least one contour point are at a distance from an edge of the central area that is less than a preset value.
[0037] In a possible implementation, the circle detection is used to filter candidate contours for which the ratio of the distance between at least one contour point coordinate and the center point coordinate to the radius value is not within a preset ratio range.
[0038] In a possible implementation, the standard deviation detection is used to filter candidate contours whose corresponding radius value differs from the mean of all the candidate contour radius values by more than a preset multiple of the standard deviation.
[0039] In a possible implementation manner, determining the positions of all target regions in the grid image based on the positions of the target regions in the central region includes:
[0040] Calculate a relative position vector and an orthogonal vector having the same modulus as the relative position vector as a unit grid vector according to position information corresponding to the positions of the two target regions closest to each other in the central region;
[0041] Fine-tuning the unit network vector according to the relative position vector between each two adjacent target area positions in the central area;
[0042] Randomly determine the coordinates of the center point of a target area as the starting position;
[0043] Determining multiple horizontal and vertical center point positions in sequence starting from the starting position according to the finely adjusted unit grid vector;
[0044] All target area positions in the grid image are marked according to each of the center point positions and the radius values of the target area positions in the center area.
[0045] According to a second aspect of the present disclosure, there is provided an image processing device, the device comprising:
[0046] An image determination module, used to determine a grid image including a plurality of target areas, each of which has the same shape and size and is arranged in a grid form;
[0047] An area determination module, used to determine the central area of the grid image;
[0048] A position identification module, used to identify the position of the target area included in the central area;
[0049] A position matching module is used to determine the positions of all target areas in the grid image based on the positions of the target areas in the central area.
[0050] In a possible implementation manner, the image determination module is further used to:
[0051] Get the image to be processed;
[0052] The effective area in the image to be processed is extracted to obtain a grid image including a plurality of target areas.
[0053] In a possible implementation manner, the image determination module is further used to:
[0054] Convert the image to be processed into a grayscale image and perform thresholding;
[0055] Performing contour recognition on the image to be processed after format conversion and thresholding, obtaining a binary mask area surrounded by the largest contour as a valid area;
[0056] The effective area is extracted from the image to be processed to obtain a grid image including a plurality of target areas.
[0057] In a possible implementation manner, the region determination module is further configured to:
[0058] intercepting a candidate region from the grid image;
[0059] The position of the coarse-grained target area in the candidate area is marked by image processing to obtain the central area of the grid image.
[0060] In a possible implementation manner, the region determination module is further configured to:
[0061] Determine the coordinates of the upper left corner of the grid image, and the width and height of the grid image;
[0062] The upper left corner coordinates of the candidate area and the width and height of the candidate area are calculated according to the preset scaling size, the width, height and upper left corner coordinates of the grid image, and the candidate area is cut out to obtain.
[0063] In a possible implementation manner, the region determination module is further configured to:
[0064] Performing Gaussian filtering on the candidate area;
[0065] The candidate area after Gaussian filtering is subjected to multi-threshold processing, and a target threshold is determined from multiple thresholds to mask the candidate area to obtain a central area with a coarse-grained target area position marked.
[0066] In a possible implementation, the location identification module is further used to:
[0067] Extracting contour information in the central area and obtaining the area wrapped by each contour;
[0068] Perform contour screening according to the area wrapped by each contour to obtain at least one candidate contour;
[0069] Determine the position information of each candidate contour;
[0070] Shape screening is performed according to the position information of each candidate contour to obtain a target contour for marking the position of the target area.
[0071] In a possible implementation manner, the candidate contour is a circular contour, and the position recognition module is further configured to:
[0072] Obtaining the contour point coordinates of the highest point, the lowest point, the leftmost point and the rightmost point of each candidate contour, and calculating the center point coordinates of the candidate contour and the radius value of the candidate contour according to the four contour point coordinates;
[0073] The corresponding position information is determined according to the contour point coordinates, the center point coordinates and the radius value of each candidate contour.
[0074] In a possible implementation, the location identification module is further used to:
[0075] Perform shape screening by performing at least one of boundary detection, circle detection and standard deviation detection based on the position information of each candidate contour, and delete the candidate contours that do not meet at least one detection requirement;
[0076] The candidate contour that passes the shape screening is determined as the target contour that marks the position of the target area.
[0077] In a possible implementation, the boundary detection is used to filter candidate contours whose coordinates of at least one contour point are at a distance from an edge of the central area that is less than a preset value.
[0078] In a possible implementation, the circle detection is used to filter candidate contours for which the ratio of the distance between at least one contour point coordinate and the center point coordinate to the radius value is not within a preset ratio range.
[0079] In a possible implementation, the standard deviation detection is used to filter candidate contours whose corresponding radius value differs from the mean of all the candidate contour radius values by more than a preset multiple of the standard deviation.
[0080] In a possible implementation, the location matching module is further used to:
[0081] Calculate a relative position vector and an orthogonal vector having the same modulus as the relative position vector as a unit grid vector according to position information corresponding to the positions of the two target regions closest to each other in the central region;
[0082] Fine-tuning the unit network vector according to the relative position vector between each two adjacent target area positions in the central area;
[0083] Randomly determine the coordinates of the center point of a target area as the starting position;
[0084] Determining multiple horizontal and vertical center point positions in sequence starting from the starting position according to the finely adjusted unit grid vector;
[0085] All target area positions in the grid image are marked according to each of the center point positions and the radius values of the target area positions in the center area.
[0086] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.
[0087] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.
[0088] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0089] In an embodiment of the present disclosure, a grid image including a plurality of target areas of the same shape and size is determined, and the target areas are arranged in a grid form in the grid image. The central area of the grid image is determined, and the positions of the target areas included in the central area are identified, and the positions of all the target areas in the grid image are determined based on the positions of the target areas in the central area. The present disclosure can greatly reduce the time for identifying the positions of the target areas by accurately identifying the positions of only some of the target areas in the middle area and then determining the positions of the target areas in the entire image, and can correct the errors caused by the identification of individual target areas, so that the results are more accurate.
[0090] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0091] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.
[0092] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown;
[0093] Figure 2 A schematic diagram showing an image to be processed according to an embodiment of the present disclosure;
[0094] Figure 3 A schematic diagram showing a central area according to an embodiment of the present disclosure;
[0095] Figure 4 A schematic diagram showing an image to be processed after image processing according to an embodiment of the present disclosure;
[0096] Figure 5 A schematic diagram showing an image processing device according to an embodiment of the present disclosure;
[0097] Figure 6 A schematic diagram showing an electronic device according to an embodiment of the present disclosure;
[0098] Figure 7 A schematic diagram showing another electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION
[0099] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0100] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0101] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.
[0102] The image processing method of the embodiment of the present disclosure can be performed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the image processing method of the embodiment of the present disclosure by a processor calling a computer-readable instruction stored in a memory.
[0103] Figure 1 FIG. 1 is a flowchart of an image processing method according to an embodiment of the present disclosure. Figure 1 As shown, an image processing method according to an embodiment of the present disclosure may include the following steps S10-S40.
[0104] Step S10: Determine a grid image including a plurality of target areas.
[0105] In a possible implementation, an electronic device determines a grid image including a plurality of target areas arranged in a grid form. Each target area has the same shape and size, and the shape of the target area may be a circle or a square. Optionally, the grid image may be obtained by capturing a grid carrying an observed sample through an image acquisition device, and the target area in the grid image may be the area where the hole carrying the observed sample is located on the grid, and the image acquisition device may be a cryo-electron microscope image. The electronic device may determine the grid image by directly capturing an image of the grid through a connected image acquisition device, or may also receive an image sent by other devices after capturing an image of the grid through an image acquisition device.
[0106] Furthermore, the image captured by the image acquisition device may still contain background areas or other areas unrelated to the web content, so the electronic device can first determine that the image captured by the image acquisition device is the image to be processed, and then further determine that the area where the web is located in the image to be processed is the effective area, and extract the effective area in the image to be processed to obtain a web image including multiple target areas. Among them, the method of extracting the effective area of the image to be processed to obtain the web image can be to first convert the image to be processed into an image in grayscale format and perform thresholding to obtain a binary mask image, and then perform contour recognition on the image to be processed after format conversion and thresholding, and obtain the binary mask area surrounded by the largest contour as the effective area. Then extract the effective area from the image to be processed to obtain a web image including multiple target areas. The threshold in the thresholding process can be automatically determined according to the image distribution, and the thresholding process and the contour recognition process distribution can be implemented by the threshold function and findContours function in the opencv library.
[0107] Figure 2 FIG. 1 is a schematic diagram showing an image to be processed according to an embodiment of the present disclosure. Figure 2 As shown, the image acquisition device acquires the image of the carrier net and obtains the image to be processed, which includes the background area and the spot noise area in addition to the carrier net area. Therefore, after acquiring the image to be processed, the electronic device can extract the binary mask area surrounded by the largest contour by format conversion and threshold processing to obtain the carrier net image.
[0108] Step S20: determining the central area of the grid image.
[0109] In a possible implementation, after determining the grid image, the electronic device intercepts the central area therein to accurately identify the position of the target area in the central area. The central area is a part of the grid image. In this embodiment, the target area (hole) in the central area can be first identified, and then the position of the target area in other areas of the grid image outside the central area can be determined according to the target area in the central area. In this way, the regularity of the position distribution of the holes in the grid is fully utilized. By identifying the local central area, it can be extended to the entire grid image, thereby improving the recognition efficiency. Among them, before accurately identifying the target area in the central area, the position of the target area can also be coarsely marked in the central area. The coarse granularity means that the marking accuracy is low, so as to first determine the approximate position and shape of the target area, and then accurately identify it, so as to further improve the efficiency of accurate identification. The process of the electronic device obtaining the central area can be to first intercept the candidate area from the grid image, and then mark the coarse-grained target area position in the candidate area through image processing to obtain the central area of the grid image. The range of the central area and the candidate area is consistent. The central area can be understood as a coarse-grained marking of the target area in the candidate area on the basis of the candidate area.
[0110] Optionally, the candidate region in the grid image can be obtained by first determining the coordinates of the upper left corner of the grid image, as well as the width and height of the grid image. Then, the coordinates of the upper left corner of the candidate region and the width and height of the candidate region are calculated according to the preset scaling size, the width, height and upper left corner coordinates of the grid image, and the candidate region is obtained by cutting. That is, the rectangular bounding box of the grid image can be calculated by determining the coordinates, width and height of the upper left corner of the grid image, and then the rectangular bounding box is scaled according to the preset scaling size to determine the coordinates of the upper left corner of the candidate region and the width and height of the candidate region, and the bounding box of the candidate region is obtained to cut the candidate region. Wherein, when the scaling size is a, the coordinates of the upper left corner of the grid image are (x and y), and the values of the width and height of the grid image are w and h respectively, the coordinates of the upper left corner of the candidate region (x0, y0) and the width and height of the candidate region (w0 and h0) can be calculated by x0=x+w*(1-a) / 2, y0=y+h*(1-a) / 2, w0=a*w, h0=a*h.
[0111] Further, after determining the candidate area, the electronic device performs coarse-grained position identification on the target area in the candidate area to obtain the central area. Among them, the process of coarse-grained labeling can be to first perform Gaussian filtering on the candidate area, then perform multi-threshold processing on the candidate area after Gaussian filtering, and determine the target threshold from multiple thresholds to mask the candidate area to obtain the central area of the coarse-grained target area position. The Gaussian filtering process is used to smooth the candidate area and reduce noise. The multi-threshold processing process is used to coarsely extract the target area position in the candidate area. The embodiment of the present disclosure selects a multi-threshold processing method to automatically give thresholds for multiple areas with different distributions to avoid erroneous extraction of polluted areas in the image. Optionally, the multi-category thresholding method can be to perform thresholding on the candidate area according to a preset number of thresholds, and the method of determining the target threshold can be to traverse all thresholds after thresholding from large to small, and determine the threshold between 10% and 80% of the total area of the positive area of the masked image as the target threshold. When the threshold that meets the above conditions is greater than one, the threshold with the largest total area of the positive area of the masked image can be selected as the target threshold.
[0112] Figure 3 A schematic diagram of a central area according to an embodiment of the present disclosure is shown. Figure 3 As shown in the figure, the left image is a candidate area obtained by cutting the middle part of the web image, which includes multiple circular target areas. The right image is a central area obtained after coarse-grained recognition of the candidate area, wherein the central area includes the coarsely marked target area position. After determining the central area, the electronic device can further accurately mark the target area position based on the coarsely marked target area position.
[0113] Step S30: Identify the location of the target area included in the central area.
[0114] In a possible implementation, after determining the central area, the electronic device performs more accurate position identification on the target area included in the central area. Optionally, the electronic device can accurately identify the target area in the central area by contour recognition. Exemplarily, the electronic device can first perform contour recognition on the central area, and then perform two screenings according to the contour area and position to obtain the position of the target area, that is, the contour information in the central area can be extracted and the area wrapped by each contour can be obtained, and then the contour screening is performed according to the area wrapped by each contour to obtain at least one candidate contour. Then the position information of each candidate contour is determined, and the shape screening is performed according to the position information of each candidate contour to obtain the target contour for marking the position of the target area. Among them, the contour recognition method can be to directly extract the coarse-grained target area position marked in the central area, and obtain the contour information corresponding to each coarse-grained target area position, that is, the contour of the coarse-grained target area is used as the initial contour. Further, the electronic device can traverse each contour to obtain the area wrapped by each contour, and perform the first round of contour screening. Then determine the position information corresponding to the candidate contours remaining after the first round of screening, and then perform the second round of shape screening according to the position information to obtain the target contour.
[0115] Optionally, the first round of screening process can be performed based on a preset pixel size. For example, contours with an area less than 5000 square pixels can be filtered out to obtain candidate contours. Furthermore, the candidate contours can be circular contours, and the position information of each candidate contour can include the coordinates of points located at the highest, lowest, leftmost and rightmost positions of the contour, as well as the radius of the circle. That is, the electronic device can determine the position information of the candidate contour by obtaining the contour point coordinates of the highest point, lowest point, leftmost point and rightmost point of each candidate contour, and calculating the center point coordinates of the candidate contour and the radius value of the candidate contour based on the four contour point coordinates. The corresponding position information is then determined based on the contour point coordinates, center point coordinates and radius value of each candidate contour. Among them, the center point coordinates can be obtained by calculating the average of the four contour point coordinates, and the radius value can be obtained based on the circle area formula S=πr 2 Calculated.
[0116] Further, after the electronic device determines the position information of the candidate contours after the first round of screening, it can perform shape screening by performing at least one of boundary detection, circle detection and standard deviation detection based on the position information of each candidate contour, delete the candidate contours that do not meet at least one pre-selected detection requirement, and then determine the candidate contours that pass the shape screening as the target contours that mark the position of the target area. Among them, boundary detection can be used to filter the candidate contours whose distance between at least one contour point coordinate and the edge of the central area is less than a preset value, that is, the electronic device can determine whether the four contour points are at the edge of the central area by the distance between the four contour points and the edge of the central area, and filter out the candidate contour in the case of the edge of the central area. Circle detection is used to filter the candidate contours whose ratio of the distance between at least one contour point coordinate and the center point coordinate to the radius value is not within a preset ratio range. In the process of performing circle detection, the electronic device can determine a preset ratio range, such as 0.75-1.25, calculate the ratio of the distance between each contour point and the center point to the radius value, and filter out the candidate contour if there is a ratio that is not within the preset ratio range. Standard deviation detection is used to filter candidate contours whose corresponding radius value differs from the mean radius value of all candidate contours by more than a preset multiple of the standard deviation. Standard deviation detection can be used to filter contours that have passed circle detection and boundary detection.
[0117] After screening the contours in the above two ways, the electronic device can determine the contours that pass the two screenings as the target contour, and determine the area included in the target contour as the target area.
[0118] Step S40: determining the positions of all target regions in the grid image based on the positions of the target regions in the central region.
[0119] In a possible implementation, after accurately identifying the target area position in the central area, the electronic device can determine the positions of other target areas in the grid image based on the identified target area positions in the central area through a grid matching algorithm. Among them, the electronic device can first determine the unit grid vector based on the identified target area position, and then execute the grid matching algorithm based on the unit grid vector to determine the positions of other target areas in the grid image. Exemplarily, the electronic device can first calculate the relative position vector based on the position information corresponding to the two closest target area positions in the central area, and the orthogonal vector with the same modulus as the relative position vector as the unit grid vector. Then fine-tune the unit grid vector according to the relative position vector between each two adjacent target area positions in the central area. Randomly determine the center point coordinates of a target area position as the starting position, and determine multiple horizontal and vertical center point positions in sequence from the starting position according to the fine-tuned unit grid vector, and finally mark all target area positions in the grid image according to the radius value of each center point position and the target area position in the central area.
[0120] Exemplarily, after the electronic device determines the target area position in the central area, it can determine the center point coordinates of the two closest target areas as (x1, y1) and (x2, y2), and calculate the relative position vector d = (x2-x1, y2-y1), and calculate the orthogonal vector d' = (y1-y2, x2-x1) with the same modulus as the relative position vector. Use d and d' as a set of initial unit grid vectors. Further, the unit grid vector is fine-tuned according to the center point coordinates of other target area positions in the central area. The fine-tuning process may include calculating the relative position vectors between the center point coordinates of all target areas in the central area. For all position vectors whose cosine values relative to the initial position vector d are greater than a preset value (such as 0.995), the corresponding unit vectors are obtained by dividing the position vector by the ratio of the vector modulus to the integer modulus of d, and the mean of these unit vectors is used to replace d as the horizontal unit grid vector; similarly, for all position vectors whose cosine values relative to d' are greater than a preset value (such as 0.995), the corresponding unit vectors are obtained by dividing the position vector by the ratio of the vector modulus to the integer modulus of d', and the mean of these unit vectors is used to replace d' as the vertical unit grid vector, thereby achieving fine-tuning of the unit network vector.
[0121] Furthermore, after obtaining the fine-tuned unit grid vector, the electronic device can obtain the coordinates of the center point of any target area in the central area, add its coordinates to the coordinates of the upper left corner of the central area, and obtain the position coordinates c = (x0, y0) of the center point in the grid image. After determining the position coordinates, the position coordinates can be used as the starting position, i is set as the horizontal axis extension multiple, j is set as the vertical axis extension multiple, i is multiplied by the horizontal unit grid vector d obtained after fine-tuning, and j is multiplied by the vertical unit grid vector d' obtained after fine-tuning, and the sum of the two is added to the starting point coordinates c to obtain a new center point coordinate of the target area. The values of i and j are initially set to 0, and the first point obtained is the starting position itself. Keep i = 0, increase j = 0, 1, 2, ... upward, repeat the above method of calculating the center point coordinates to obtain new center point coordinates, until the calculated center point coordinates are no longer within the grid image, and record the obtained center point coordinates. Similarly, decrement j = -1, -2, -3, ... downward, repeat the calculation to obtain the new center point coordinates, until the calculated center point coordinates are not within the grid image, and record the obtained center point coordinates. Similarly, you can also increase i = 1, 2, 3, ... upward, repeat the above calculation of increasing and decreasing j for each i, and record the new center point coordinates; increase i = -1, -2, -3, ... downward, repeat the above calculation of increasing and decreasing j for each i, and record the new center point coordinates. The electronic device counts all the obtained center point coordinates to obtain the center position of all circular target areas in the grid image, and further marks all target area positions in the grid image according to the radius value and center point coordinates of the target area position in the center area.
[0122] Figure 4 FIG. 1 is a schematic diagram showing an image to be processed after image processing according to an embodiment of the present disclosure. Figure 4 As shown, the electronic device can mark all target areas in the grid image through the above-mentioned grid matching algorithm.
[0123] Based on the above technical features, the disclosed embodiment only accurately identifies the position of part of the target area in the middle area, and then determines the position of the target area in the entire image. Since accurate position identification is only performed on part of the area, this identification method greatly reduces the time for identifying the position of the target area, and can correct the errors caused by the identification of individual target areas, making the results more accurate. At the same time, the disclosed embodiment can adapt to pictures of various field of view ratios, and the application of Gaussian filtering and multi-category thresholding in the process of image processing can enable the algorithm to combat various types of noise and pollution, thereby enhancing the robustness of the algorithm.
[0124] Figure 5 FIG. 1 is a schematic diagram showing an image processing device according to an embodiment of the present disclosure. Figure 5 As shown, the image processing device of the embodiment of the present disclosure may include:
[0125] An image determination module 50, for determining a grid image including a plurality of target areas, each of which has the same shape and size and is arranged in a grid form;
[0126] An area determination module 51 is used to determine the central area of the grid image;
[0127] A position identification module 52, used to identify the position of the target area included in the central area;
[0128] The position matching module 53 is used to determine the positions of all target areas in the grid image based on the positions of the target areas in the central area.
[0129] In a possible implementation, the image determination module 50 is further configured to:
[0130] Get the image to be processed;
[0131] The effective area in the image to be processed is extracted to obtain a grid image including a plurality of target areas.
[0132] In a possible implementation, the image determination module 50 is further configured to:
[0133] Convert the image to be processed into a grayscale image and perform thresholding;
[0134] Performing contour recognition on the image to be processed after format conversion and thresholding, obtaining a binary mask area surrounded by the largest contour as a valid area;
[0135] The effective area is extracted from the image to be processed to obtain a grid image including a plurality of target areas.
[0136] In a possible implementation, the region determination module 51 is further configured to:
[0137] intercepting a candidate region from the grid image;
[0138] The position of the coarse-grained target area in the candidate area is marked by image processing to obtain the central area of the grid image.
[0139] In a possible implementation, the region determination module 51 is further configured to:
[0140] Determine the coordinates of the upper left corner of the grid image, and the width and height of the grid image;
[0141] The upper left corner coordinates of the candidate area and the width and height of the candidate area are calculated according to the preset scaling size, the width, height and upper left corner coordinates of the grid image, and the candidate area is cut out to obtain.
[0142] In a possible implementation, the region determination module 51 is further configured to:
[0143] Performing Gaussian filtering on the candidate area;
[0144] The candidate area after Gaussian filtering is subjected to multi-threshold processing, and a target threshold is determined from multiple thresholds to mask the candidate area to obtain a central area with a coarse-grained target area position marked.
[0145] In a possible implementation, the location identification module 52 is further configured to:
[0146] Extracting contour information in the central area and obtaining the area wrapped by each contour;
[0147] Perform contour screening according to the area wrapped by each contour to obtain at least one candidate contour;
[0148] Determine the position information of each candidate contour;
[0149] Shape screening is performed according to the position information of each candidate contour to obtain a target contour for marking the position of the target area.
[0150] In a possible implementation, the candidate contour is a circular contour, and the position recognition module 52 is further configured to:
[0151] Obtaining the contour point coordinates of the highest point, the lowest point, the leftmost point and the rightmost point of each candidate contour, and calculating the center point coordinates of the candidate contour and the radius value of the candidate contour according to the four contour point coordinates;
[0152] The corresponding position information is determined according to the contour point coordinates, the center point coordinates and the radius value of each candidate contour.
[0153] In a possible implementation, the location identification module 52 is further configured to:
[0154] Perform shape screening by performing at least one of boundary detection, circle detection and standard deviation detection based on the position information of each candidate contour, and delete the candidate contours that do not meet at least one detection requirement;
[0155] The candidate contour that passes the shape screening is determined as the target contour that marks the position of the target area.
[0156] In a possible implementation, the boundary detection is used to filter candidate contours whose coordinates of at least one contour point are at a distance from an edge of the central area that is less than a preset value.
[0157] In a possible implementation, the circle detection is used to filter candidate contours for which the ratio of the distance between at least one contour point coordinate and the center point coordinate to the radius value is not within a preset ratio range.
[0158] In a possible implementation, the standard deviation detection is used to filter candidate contours whose corresponding radius value differs from the mean of all the candidate contour radius values by more than a preset multiple of the standard deviation.
[0159] In a possible implementation, the location matching module 53 is further configured to:
[0160] Calculate a relative position vector and an orthogonal vector having the same modulus as the relative position vector as a unit grid vector according to position information corresponding to the positions of the two target regions closest to each other in the central region;
[0161] Fine-tuning the unit network vector according to the relative position vector between each two adjacent target area positions in the central area;
[0162] Randomly determine the coordinates of the center point of a target area as the starting position;
[0163] Determining multiple horizontal and vertical center point positions in sequence starting from the starting position according to the finely adjusted unit grid vector;
[0164] All target area positions in the grid image are marked according to each of the center point positions and the radius values of the target area positions in the center area.
[0165] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0166] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.
[0167] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.
[0168] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0169] Figure 6 A schematic diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0170] Reference Figure 6 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .
[0171] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.
[0172] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0173] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.
[0174] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0175] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.
[0176] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.
[0177] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
[0178] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0179] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0180] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.
[0181] Figure 7 FIG. 1 is a schematic diagram of another electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.
[0182] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.
[0183] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.
[0184] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.
[0185] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.
[0186] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0187] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0188] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0189] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0190] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0191] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.
[0192] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. An image processing method, characterized in that: The method comprises: Determining a grid image including a plurality of target areas, each of the target areas having the same shape and size and arranged in a grid format; Determining a central area of the grid image; Identify a target area position included in the central area; The positions of all target regions in the grid image are determined based on the positions of the target regions in the central region.
2. The method according to claim 1, characterized in that The determining of the grid image including the plurality of target areas comprises: Get the image to be processed; The effective area in the image to be processed is extracted to obtain a grid image including a plurality of target areas.
3. The method according to claim 2, characterized in that The step of extracting the effective area in the image to be processed to obtain a grid image including a plurality of target areas includes: Convert the image to be processed into a grayscale image and perform thresholding; Performing contour recognition on the image to be processed after format conversion and thresholding, obtaining a binary mask area surrounded by the largest contour as a valid area; The effective area is extracted from the image to be processed to obtain a grid image including a plurality of target areas.
4. The method according to any one of claims 1 to 3, characterized in that: The determining the central area of the grid image includes: intercepting a candidate region from the grid image; The position of the coarse-grained target area in the candidate area is marked by image processing to obtain the central area of the grid image.
5. The method according to claim 4, characterized in that The extracting the candidate area from the grid image comprises: Determine the coordinates of the upper left corner of the grid image, and the width and height of the grid image; The upper left corner coordinates of the candidate area and the width and height of the candidate area are calculated according to the preset scaling size, the width, height and upper left corner coordinates of the grid image, and the candidate area is cut out to obtain.
6. The method according to claim 4 or 5, characterized in that: The method of marking the position of the coarse-grained target area in the candidate area by image processing to obtain the central area of the grid image includes: Performing Gaussian filtering on the candidate area; The candidate area after Gaussian filtering is subjected to multi-threshold processing, and a target threshold is determined from multiple thresholds to mask the candidate area to obtain a central area with a coarse-grained target area position marked.
7. The method according to any one of claims 1 to 6, characterized in that: The identifying the location of the target area included in the central area comprises: Extracting contour information in the central area and obtaining the area wrapped by each contour; Perform contour screening according to the area wrapped by each contour to obtain at least one candidate contour; Determine the position information of each candidate contour; Shape screening is performed according to the position information of each candidate contour to obtain a target contour for marking the position of the target area.
8. The method according to claim 7, characterized in that The candidate contour is a circular contour, and determining the position information of each candidate contour includes: Obtaining the contour point coordinates of the highest point, the lowest point, the leftmost point and the rightmost point of each candidate contour, and calculating the center point coordinates of the candidate contour and the radius value of the candidate contour according to the four contour point coordinates; The corresponding position information is determined according to the contour point coordinates, the center point coordinates and the radius value of each candidate contour.
9. The method according to claim 7 or 8, characterized in that: The step of performing shape screening according to the position information of each candidate contour to obtain a target contour for marking the position of the target area includes: Perform shape screening by performing at least one of boundary detection, circle detection and standard deviation detection based on the position information of each candidate contour, and delete the candidate contours that do not meet at least one detection requirement; The candidate contour that passes the shape screening is determined as the target contour that marks the position of the target area.
10. The method according to claim 9, characterized in that The boundary detection is used to filter candidate contours whose coordinates of at least one contour point are less than a preset value from the edge of the central area.
11. The method according to claim 9 or 10, characterized in that: The circle detection is used to filter candidate contours whose ratio of the distance between at least one contour point coordinate and the center point coordinate to the radius value is not within a preset ratio range.
12. The method according to any one of claims 9 to 11, characterized in that: The standard deviation detection is used to filter candidate contours whose corresponding radius value differs from the mean radius value of all the candidate contours by a greater than a preset multiple standard deviation.
13. The method according to any one of claims 1 to 12, characterized in that: The determining the positions of all target areas in the grid image based on the positions of the target areas in the central area comprises: Calculate a relative position vector and an orthogonal vector having the same modulus as the relative position vector as a unit grid vector according to position information corresponding to the positions of the two target regions closest to each other in the central region; Fine-tuning the unit network vector according to the relative position vector between each two adjacent target area positions in the central area; Randomly determine the coordinates of the center point of a target area as the starting position; Determining multiple horizontal and vertical center point positions in sequence starting from the starting position according to the finely adjusted unit grid vector; All target area positions in the grid image are marked according to each of the center point positions and the radius values of the target area positions in the center area.
14. An image processing device, characterized in that: The device comprises: An image determination module, used to determine a grid image including a plurality of target areas, each of which has the same shape and size and is arranged in a grid form; An area determination module, used to determine the central area of the grid image; A position identification module, used to identify the position of the target area included in the central area; A position matching module is used to determine the positions of all target areas in the grid image based on the positions of the target areas in the central area.
15. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 13 when executing the instructions stored in the memory.
16. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 13 is implemented.