Image processing method, system, electronic device and storage medium

Abstract

A method, system, electronic device and storage medium for image imaging, relating to the field of image processing. In this method, image data is collected by a preset sensor, and it is determined whether a pattern scanning instruction issued by a user is received; when a pattern scanning instruction issued by a user is not received, the collected image data is imaged and processed, and the offset of the first pattern after imaging from the preset center position is calculated, and whether the preset rapid positioning condition is met according to the offset; when the preset rapid positioning condition is met, the target area is determined according to the historical appearance position information of the first pattern, and feature comparison is performed in the target area to locate the first pattern; the first pattern is corrected according to the offset. The implementation of the technical solution provided by the present application improves the accuracy and efficiency of image processing, especially the pattern recognition and positioning capabilities in complex environments.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to an image imaging processing method, system, electronic device and storage medium. Background Art

[0002] Image processing technology has developed rapidly in recent years and is widely used in industrial inspection, medical imaging, security monitoring and other fields.

[0003] However, existing image processing technologies still have some shortcomings in practical applications. Especially in complex environments, due to factors such as lighting changes and the presence of obstructions, traditional fixed template matching and feature point-based detection algorithms are prone to false detection and missed detection. In addition, existing image processing technologies lack an effective correction mechanism for image offset and rotation, resulting in low positioning accuracy and unable to meet the needs of high-precision application scenarios.

[0004] Therefore, there is a need for an image imaging processing method that can perform pattern recognition and positioning in complex environments. Summary of the invention

[0005] The present application provides an image processing method, system, electronic device and storage medium, which improve the accuracy and efficiency of image processing, especially the pattern recognition and positioning capabilities in complex environments.

[0006] In a first aspect of the present application, a method for processing image imaging is provided, which is applied to an image imaging processing platform, and the method comprises:

[0007] Collect image data through a preset sensor and determine whether a pattern scanning instruction issued by a user is received;

[0008] When no pattern scanning instruction is received from the user, imaging processing is performed on the collected image data, an offset of the imaged first pattern from a preset center position is calculated, and whether a preset fast positioning condition is met according to the offset;

[0009] When a preset fast positioning condition is met, determining a target area according to the historical occurrence position information of the first pattern, and performing feature comparison in the target area to locate the first pattern;

[0010] The first pattern is skew-corrected according to the offset.

[0011] Optionally, the processing method further includes:

[0012] When receiving a pattern scanning instruction from the user, display the currently acquired image data and wait for the user to confirm the second pattern to be tracked;

[0013] After the user confirms, the second pattern is feature extracted by the preset sensor, and the extracted feature information of the second pattern is classified and stored.

[0014] Optionally, the calculating an offset of the imaged first pattern from a preset center position, and determining whether a preset rapid positioning condition is met according to the offset includes:

[0015] Acquire characteristic information of the first pattern, and determine a center point of the first pattern and a preset fast positioning condition according to the characteristic information;

[0016] Calculating a first distance and a second distance between the center point and the preset center position, and calculating a first offset between the center point and the preset center position according to the first distance and the second distance, wherein the first distance is a distance in a first direction, and the second distance is a distance in a second direction, and the first direction is perpendicular to the second direction;

[0017] The first offset is compared with an offset threshold in the preset fast positioning condition to determine whether the preset fast positioning condition is met.

[0018] Optionally, determining the target area according to the historical appearance position information of the first pattern includes:

[0019] Divide the area to be searched into multiple first preset areas according to a first preset rule, and determine whether the frequency of the first pattern appearing alone in any first preset area exceeds a first threshold according to the historical appearance position information, wherein the first preset rule is to draw a straight line perpendicular to the edge according to the equally divided points of all edges of the area to be searched, and divide the area to be searched according to the straight line and the intersection of the straight lines;

[0020] When the frequency of the first pattern appearing alone in the first preset target area exceeds the first threshold, the first preset target area is determined as the target area;

[0021] When the frequency of the first pattern appearing alone in any first preset area does not exceed the first threshold, determining whether the sum of the frequencies of the first pattern appearing in two adjacent first preset areas exceeds a second threshold;

[0022] When the frequency of the first pattern appearing in the second preset target area and the third preset target area exceeds the second threshold, the second preset target area and the third preset target area are determined as target areas, and the second preset target area is adjacent to the third preset target area.

[0023] Optionally, determining the target area according to the historical appearance position information of the first pattern further includes:

[0024] When the sum of the frequencies of the first pattern appearing in two adjacent first preset areas does not exceed a second threshold, the area to be searched is divided into a plurality of second preset areas according to a second preset rule, wherein the second preset rule is to draw straight lines perpendicular to the target edge according to the equally divided points of the target edge of the area to be searched, and divide the area to be searched according to the straight lines;

[0025] A second preset area where the first pattern appears most frequently is determined as a target area.

[0026] Optionally, performing feature comparison in the target area to locate the first pattern includes:

[0027] combining features of a first pixel row and a second pixel row of the first pattern into a target feature, wherein the first pixel row is a pixel row in which pixels of the first pattern first appear along a third direction, and the second pixel row is a pixel row in which pixels of the first pattern last appear along the third direction;

[0028] Extracting image features of all patterns in the target area, and comparing the image features with the target features;

[0029] A pattern in the target area whose comparison similarity exceeds a similarity threshold is determined as the first pattern, and position information of the first pattern is recorded.

[0030] Optionally, combining features of the first pixel row and the second pixel row of the first pattern into a target feature comprises:

[0031] Acquire a first feature of the first pixel row of the first pattern, and acquire a second feature of the second pixel row of the first pattern, wherein the first feature includes a first number and a first position of pixels, and the second feature includes a second number and a second position of pixels;

[0032] A first characteristic relationship between the first quantity and the second quantity is established, a second characteristic relationship between the first position and the second position is established, and the first characteristic relationship and the second characteristic relationship are combined into a target characteristic.

[0033] In a second aspect of the present application, an image processing system is provided, comprising an acquisition module, a calculation module, a positioning module and a correction module, wherein:

[0034] An acquisition module, configured to acquire image data through a preset sensor and determine whether a pattern scanning instruction issued by a user is received;

[0035] A calculation module, configured to perform imaging processing on the collected image data when no pattern scanning instruction issued by the user is received, calculate the offset of the first pattern after imaging from the preset center position, and determine whether the preset fast positioning condition is met according to the offset;

[0036] a positioning module configured to determine a target area according to historical occurrence position information of the first pattern when a preset fast positioning condition is met, and perform feature comparison in the target area to locate the first pattern;

[0037] A correction module is configured to correct the first pattern according to the offset.

[0038] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0039] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0040] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0041] 1. When the system does not receive the pattern scanning instruction issued by the user, it can automatically process the collected image data and calculate the offset of the pattern. This automated process reduces manual intervention and improves processing efficiency;

[0042] 2. Through the preset fast positioning conditions, it can intelligently determine when to enable the fast positioning mechanism. When the conditions are met, the system uses the historical location information of the pattern to narrow the search range, thereby performing feature comparison in the target area and achieving fast positioning. This method significantly improves the positioning speed, especially when processing a large amount of image data;

[0043] 3. Based on the calculated offset, the system combines the historical location information of the pattern to more accurately determine the target area. This positioning method that combines historical data and real-time data has higher accuracy and stability than single real-time data positioning;

[0044] 4. Feature comparison is performed in the target area to further ensure the accuracy of positioning. Feature comparison can identify the target pattern and determine its exact location by comparing the feature points of the pattern with the feature points in the preset feature library;

[0045] 5. Based on the calculated offset, the system can perform deskew processing on the pattern. The deskew operation can adjust the position of the pattern to make it closer to the preset center position, thereby improving the overall layout and visual effect of the image. Deskew processing also helps to reduce image distortion and blur caused by pattern position offset, and improve image clarity and readability. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flowchart of the image imaging processing method disclosed in the embodiment of the present application;

[0047] Figure 2 is another flowchart of the image imaging processing method disclosed in the embodiment of the present application;

[0048] Figure 3 It is a characteristic schematic diagram of a triangle, a rectangle and a straight line disclosed in the embodiment of the present application;

[0049] Figure 4 is a schematic diagram of the target pattern disclosed in the embodiment of the present application in different preset areas;

[0050] Figure 5 It is a module schematic diagram of the image processing system disclosed in the embodiment of the present application;

[0051] Figure 6 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0052] Explanation of the reference numerals: 501, acquisition module; 502, calculation module; 503, positioning module; 504, correction module; 601, processor; 602, communication bus; 603, user interface; 604, network interface; 605, memory. DETAILED DESCRIPTION

[0053] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0054] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0055] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0056] This embodiment discloses an image processing method, which is applied to an image processing platform. Figure 1 is a flow chart of the image processing method disclosed in the embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0057] S110, collecting image data through a preset sensor, and determining whether a pattern scanning instruction issued by a user is received;

[0058] Figure 2 is another flow chart of the image processing method disclosed in the embodiment of the present application, combined with Figure 2 The embodiments of the present application are described below. Figure 2 As shown in the figure, the preset sensor is a CCD (Charge Coupled Device) / CMOS (Complementary Metal Oxide Semiconductor) sensor. The CCD / CMOS sensor will be initialized after power-on to ensure that the sensor can work normally and is ready for subsequent image data acquisition. After the initialization is completed, the sensor immediately enters the signal acquisition stage. In this stage, the sensor will continuously collect image data from its field of view. These image data are the basis for subsequent processing and analysis. At the same time, the sensor will also cyclically scan whether the user has performed specific operations to trigger the scanning step of the target pattern. The "user operation" mentioned here usually refers to the instructions issued by the user to the system in some way (such as button pressing, touch screen operation, voice command, etc.). During the whole process, the sensor is always collecting image data. The system is constantly monitoring whether the user has issued a pattern scanning instruction. The two are carried out in parallel and do not interfere with each other. Once the system receives the user's pattern scanning instruction, it will immediately extract the part related to the target pattern from the continuously collected image data and perform subsequent processing and analysis.

[0059] Optionally, the processing method further includes:

[0060] When receiving a pattern scanning instruction from the user, display the currently acquired image data and wait for the user to confirm the second pattern to be tracked;

[0061] After the user confirms, the second pattern is feature extracted by the preset sensor, and the extracted feature information of the second pattern is classified and stored.

[0062] When the user needs to scan and track a specific pattern through the system, a pattern scanning instruction will be issued. After receiving this instruction, the system is ready for subsequent operations. The system will immediately display the image data of the environment or target area collected by the CCD or CMOS sensor in real time to the user so that the user can clearly see the picture captured by the current sensor. The user selects and confirms the pattern to be tracked based on the displayed image data, that is, the second pattern. Through the preset sensor (that is, CCD or CMOS sensor), the system extracts features of the second pattern confirmed by the user. The feature extraction process may involve image preprocessing, edge detection, corner detection, texture analysis and other technologies to obtain the uniqueness and distinguishability of the pattern. The extracted second pattern feature information is classified and stored by the system. The storage method may involve a database, file system or other data storage technology to ensure the integrity and accessibility of the feature information. The classified and stored feature information will be used for subsequent target tracking, image matching, correction and other operations.

[0063] By displaying the currently acquired image data, users can intuitively see the images captured by the sensor, making it easier to select and confirm the second pattern to be tracked. This instant feedback mechanism improves the convenience and accuracy of user operations. After the user confirms the second pattern to be tracked, the system will extract features of the pattern. Feature extraction is a key step in image processing and computer vision, which can extract representative and distinguishing information from the image. By accurately extracting the features of the second pattern, the system can more accurately identify and track the pattern and improve recognition accuracy. The feature information of the extracted second pattern will be classified and stored. This storage method not only saves storage space, but also facilitates the subsequent retrieval and use of feature information. Classification storage also helps the system build a feature library, providing strong support for subsequent image matching and recognition. By classifying and storing the feature information of different patterns, the system can support the tracking and recognition of multiple patterns. This is of great significance for application scenarios that require simultaneous monitoring of multiple targets (such as intelligent monitoring, autonomous driving, etc.). This processing method allows users to select and confirm the pattern to be tracked as needed, thereby improving the flexibility of the system. Users can adjust the tracking target according to actual needs, making the system more adaptable to different application scenarios and needs.

[0064] Figure 3 is a characteristic schematic diagram of a triangle, a rectangle and a straight line disclosed in the embodiment of the present application, such as Figure 3 As shown, the pixels of the triangle pattern will change in the vertical columns of pixels. From pixel row 1 to row 4, the vertical pixels expand to both sides with the triangle vertex as the center; the pixels of the rectangular pattern will also change in the vertical columns of pixels. From pixel row 1 to row 4, the vertical pixels exist as multiple pixels in pixel row 1, and then the pixels of the two columns go straight down from row 1 to row 4; the position of the pixels of the straight line pattern in the vertical columns of pixels remains unchanged, and goes straight down from pixel row 1 to row 4.

[0065] S120, when no pattern scanning instruction is received from the user, performing imaging processing on the collected image data, calculating the offset of the imaged first pattern from the preset center position, and determining whether a preset fast positioning condition is met according to the offset;

[0066] The sensor will perform imaging processing on the collected image data. This process may include steps such as denoising, contrast enhancement, and distortion correction to ensure that the image quality is high enough for subsequent analysis and processing. After imaging processing, the sensor will try to identify the first pattern in the image. When the first pattern is identified, the sensor will calculate the offset of the first pattern relative to the preset center position. This offset is usually obtained by comparing the coordinates of the center point of the pattern with the preset center position. The preset center position can be the center position of the sensor or the center position of the area to be retrieved, or some other specific position, which can be set differently according to the specific scene. The sensor will determine whether the preset fast positioning condition is met based on the calculated offset. For example, only when the offset is within a certain range (for example, the offset is less than a threshold) and the stability or frequency of the position information reaches or exceeds the preset threshold, the system will enable the memory method for fast positioning. If the offset is too large or the position information is unstable, the current position information may be considered unreliable, and the memory method will not be enabled.

[0067] Optionally, the calculating an offset of the imaged first pattern from a preset center position, and determining whether a preset rapid positioning condition is met according to the offset includes:

[0068] Acquire characteristic information of the first pattern, and determine a center point of the first pattern and a preset fast positioning condition according to the characteristic information;

[0069] Calculating a first distance and a second distance between the center point and the preset center position, and calculating a first offset between the center point and the preset center position according to the first distance and the second distance, wherein the first distance is a distance in a first direction, and the second distance is a distance in a second direction, and the first direction is perpendicular to the second direction;

[0070] The first offset is compared with an offset threshold in the preset fast positioning condition to determine whether the preset fast positioning condition is met.

[0071] Acquire feature information of the first pattern after imaging. These feature information may be the edge, corner point, centroid, etc. of the pattern, which can accurately reflect the shape and position of the pattern. Based on these feature information, the system can further determine the center point of the first pattern. At the same time, the system also needs to understand the preset fast positioning conditions, which may include the maximum allowable offset, positioning accuracy requirements, etc. The system needs to calculate the distance between the center point and the preset center position. In order to more accurately describe the offset, the system usually considers the distance in two directions: horizontal direction (first direction) and vertical direction (second direction). These two directions are perpendicular, ensuring that the calculation of the offset can fully cover the movement of the pattern on the two-dimensional plane. First distance: refers to the distance between the center point and the preset center position in the horizontal direction. This distance reflects the offset of the pattern in the horizontal direction. Second distance: refers to the distance between the center point and the preset center position in the vertical direction. This distance reflects the offset of the pattern in the vertical direction. Based on the first distance and the second distance, the offset between the center point and the preset center position can be calculated. This offset is obtained by combining the first distance and the second distance. On the two-dimensional plane, this offset can be regarded as a vector, which has both magnitude and direction. The system needs to compare the calculated offset with the offset threshold in the preset fast positioning condition. This threshold is set according to application requirements and represents the maximum offset that the system can accept. If the calculated offset is less than or equal to the offset threshold, the system considers that the current position meets the preset fast positioning condition; if the offset is greater than the offset threshold, the system may need to cooperate with the re-edge / line search algorithm to re-locate the target pattern.

[0072] By obtaining the characteristic information of the first pattern and determining its center point accordingly, it can be ensured that the calculated offset is based on the actual position of the pattern. The distance between the calculated center point and the preset center position in the first direction and the second direction (usually horizontal and vertical directions), namely the first distance and the second distance, can fully reflect the offset of the pattern in all directions. The first offset calculated according to the first distance and the second distance is a vector value, which includes both the size of the offset and the direction of the offset, thereby improving the accuracy of positioning. The offset threshold in the preset fast positioning condition can be flexibly set according to the actual application scenario. For example, in a scenario where high-precision positioning is required, the threshold can be set to a smaller value; while in a scenario where the positioning accuracy is not required to be high, the threshold can be set to a larger value. By comparing the calculated first offset with the offset threshold, it is convenient to determine whether the current pattern position meets the requirements of fast positioning. This process is not only applicable to the positioning of static patterns, but also to the tracking of dynamic patterns. By calculating the offset in real time and comparing it with the threshold, the system can dynamically adjust its working state to adapt to changes in the pattern position. In addition, this process can also be combined with other image processing algorithms, such as image registration, image recognition, etc., to further improve the performance and adaptability of the system.

[0073] S130: when a preset fast positioning condition is met, determining a target area according to historical occurrence position information of the first pattern, and performing feature comparison in the target area to locate the first pattern;

[0074] When the system detects that the offset between the current position of the first pattern and the preset center position is less than or equal to the preset offset threshold, it is considered that the rapid positioning condition is met. The system records all the position information of the first pattern in the past period of time, and these position information form a historical data set. By analyzing this historical data set, the system can identify the area where the first pattern appears most frequently, or certain regularities of its movement trajectory (such as periodicity, trend, etc.). Based on the analysis of the historical position information, the system can predict the area where the first pattern is most likely to appear in the next period of time, and this area is the target area. The target area is usually much smaller than the entire search space, so feature matching in this smaller area will greatly improve the efficiency of positioning. After the target area is determined, the system will perform feature matching on the first pattern in this area. Feature matching refers to matching the image features in the target area with the known features of the first pattern to find the most similar position. Feature matching can be based on various image features, such as edges, corners, textures, colors, etc. When the image that best matches the features of the first pattern is found in the target area, the system considers that the first pattern has been successfully located.

[0075] Optionally, determining the target area according to the historical appearance position information of the first pattern includes:

[0076] Divide the area to be searched into multiple first preset areas according to a first preset rule, and determine whether the frequency of the first pattern appearing alone in any first preset area exceeds a first threshold according to the historical appearance position information, wherein the first preset rule is to draw a straight line perpendicular to the edge according to the equally divided points of all edges of the area to be searched, and divide the area to be searched according to the straight line and the intersection of the straight lines;

[0077] When the frequency of the first pattern appearing alone in the first preset target area exceeds the first threshold, the first preset target area is determined as the target area;

[0078] When the frequency of the first pattern appearing alone in any first preset area does not exceed the first threshold, determining whether the sum of the frequencies of the first pattern appearing in two adjacent first preset areas exceeds a second threshold;

[0079] When the frequency of the first pattern appearing in the second preset target area and the third preset target area exceeds the second threshold, the second preset target area and the third preset target area are determined as target areas, and the second preset target area is adjacent to the third preset target area.

[0080] The area to be searched is divided according to a first preset rule. The first preset rule is based on the equal division points of all edges of the area to be searched, and straight lines perpendicular to the edges are generated from these equal division points. These straight lines and their intersections divide the area to be searched into multiple first preset areas. Figure 4 is a schematic diagram of the target pattern disclosed in the embodiment of the present application in different preset areas, for example, Figure 4 In the example, the visual field can be divided into four areas 1, 2, 3, and 4 by determining the bisection point on the edge. According to the historical occurrence position information, determine whether the frequency of the first pattern appearing alone in any first preset area exceeds a first threshold (for example, 80%). The first threshold is a preset value used to determine whether the frequency of the first pattern appearing in a certain area is high enough to make the area a priority area for subsequent retrieval. If the frequency of the first pattern appearing alone in a certain first target preset area exceeds the first threshold, then the first target preset area is determined as the target area. This means that in subsequent retrievals, feature comparison will be performed preferentially in this area. For example, Figure 4The triangle in the lower left image appears on the right side of the field of view, and the pixels are distributed in area 2. After reaching the set threshold, this memorized position can be used to prioritize area 2 for feature comparison with the target pattern in subsequent corrections. If the frequency of the first pattern appearing alone in any first preset area does not exceed the first threshold, then it is necessary to consider whether the first pattern often appears in two adjacent first preset areas. At this time, determine whether the sum of the frequencies of the first pattern appearing in two adjacent first preset areas exceeds the second threshold (for example, 70%). This second threshold is also a preset value, which is used to determine the frequency of the first pattern appearing simultaneously in two adjacent areas. If the sum of the frequencies of the first pattern appearing in a second target preset area and an adjacent third target preset area exceeds the second threshold, then these two adjacent areas (the second target preset area and the third target preset area) are determined as target areas. For example, Figure 4 The triangle in the upper left corner image appears on the left side of the field of view, and the pixels are distributed in areas 1 and 3. A portion of the memory is set aside to store information about the location where the pattern appears. The cumulative appearance of the triangle in areas 1 and 3 reaches 75%. This memorized location can be used to prioritize areas 1 and 3 for feature comparison with the target pattern in subsequent corrections.

[0081] By dividing the area to be searched into a plurality of first preset areas, and judging the frequency of occurrence of the first pattern in each area based on the historical occurrence position information, the high-frequency occurrence area of ​​the first pattern can be quickly determined. When the frequency of the first pattern appearing alone in a first target preset area exceeds the first threshold, the area is directly determined as the target area, so that in subsequent retrievals, feature comparison can be preferentially performed in these areas, thereby improving the retrieval efficiency. The embodiment of the present application not only considers the case where the first pattern appears alone in a certain area, but also considers the case where it appears in two adjacent areas. When the sum of the frequency of occurrence of the first pattern in two adjacent areas exceeds the second threshold, the two areas are jointly determined as the target area. This processing method can more comprehensively capture the movement law and position preference of the first pattern, thereby improving the accuracy in subsequent positioning. The first preset rule, the first threshold and the second threshold can all be adjusted according to actual conditions. For example, these parameters can be dynamically adjusted according to the distribution of the historical occurrence position information of the first pattern. This flexibility can adapt to changes in different scenarios and needs, so that it has stronger applicability in practical applications. By determining the target area, unnecessary calculations can be reduced in subsequent retrievals. Because only feature comparison needs to be performed in the target area, and the entire area to be searched does not need to be traversed. This not only saves computing resources, but also improves the response speed of the system.

[0082] Optionally, determining the target area according to the historical appearance position information of the first pattern further includes:

[0083] When the sum of the frequencies of the first pattern appearing in two adjacent first preset areas does not exceed a second threshold, the area to be searched is divided into a plurality of second preset areas according to a second preset rule, wherein the second preset rule is to draw straight lines perpendicular to the target edge according to the equally divided points of the target edge of the area to be searched, and divide the area to be searched according to the straight lines;

[0084] A second preset area where the first pattern appears most frequently is determined as a target area.

[0085] When the frequency of the first pattern appearing alone in any first preset area does not exceed the first threshold, and the sum of its frequencies appearing in two adjacent first preset areas does not exceed the second threshold, it means that the division method of the first preset rule is not applicable. In this case, a more sophisticated division method, namely the second preset rule, is required. This rule is to draw straight lines perpendicular to the target edge (an edge) of the area to be searched based on the equal points of the target edge, and then divide the area to be searched based on these straight lines. Figure 4 As shown in the figure, the light-colored dotted lines in the two figures on the right are two straight lines perpendicular to the target edge. In actual situations, nine equal points can be determined to divide the area to be searched into ten areas, each area represents 10% pixels, that is, the first area from left to right represents the 0-10% pixel area, the second area represents the 10%-20% pixel area, and so on. Figure 4 As shown, the triangle in the upper right corner image appears more frequently in the 40-60% area of ​​the full field of view 0-100% pixels, and therefore, the 40-60% pixel area can be used as the target area.

[0086] When the sum of the frequencies of the first pattern in two adjacent first preset areas does not exceed the second threshold, it means that the target area cannot be determined according to the first preset rule. At this time, by re-dividing the area to be retrieved according to the second preset rule, the division can be further refined so that the range of each area is smaller, so that the position information of the first pattern can be captured more accurately. In practical applications, the appearance position of the first pattern may be affected by various factors, such as illumination changes, noise interference, object movement, etc. These factors may cause the appearance position of the first pattern to be relatively scattered or difficult to predict. By introducing the second preset rule and the re-division strategy, this complex situation can be adapted to improve the flexibility and accuracy of the target area determination. After re-dividing the area to be retrieved, by counting the appearance frequency of the first pattern in each second preset area, the area with the highest appearance frequency can be determined as the target area. In this way, in subsequent retrieval or positioning tasks, feature comparison or search can be performed preferentially in this area, thereby optimizing the allocation of computing resources and improving processing efficiency. By introducing the second preset rule and the re-division strategy, the robustness of the system can be improved to a certain extent. Even if the appearance position of the first pattern is disturbed or changed by external factors, the system can adapt to this change by re-dividing and counting the frequency and determine the new target area. The implementation of this supplementary strategy is relatively simple, and only requires adding some additional partitioning and statistical steps to the original strategy. This makes the strategy easy to integrate and expand with other image processing or machine vision algorithms, thus having a wider application prospect in practical applications.

[0087] Optionally, performing feature comparison in the target area to locate the first pattern includes:

[0088] combining features of a first pixel row and a second pixel row of the first pattern into a target feature, wherein the first pixel row is a pixel row in which pixels of the first pattern first appear along a third direction, and the second pixel row is a pixel row in which pixels of the first pattern last appear along the third direction;

[0089] Extracting image features of all patterns in the target area, and comparing the image features with the target features;

[0090] A pattern in the target area whose comparison similarity exceeds a similarity threshold is determined as the first pattern, and position information of the first pattern is recorded.

[0091] The first pixel row is the row where the first pattern first appears in a certain direction (e.g. vertical direction, called the third direction) in the image. In other words, it is the top edge of the first pattern in the vertical direction. The second pixel row is the row where the first pattern last appears in the same direction (the third direction) in the image. In other words, it is the bottom edge of the first pattern in the vertical direction. The features of the first pixel row and the second pixel row are combined to form a unique feature set, called the target feature. This feature set can contain a variety of information such as pixel value, color, texture, shape, etc., depending on the feature extraction method selected. Image features are extracted for all visible patterns in the target area. These features should be of the same type and dimension as the target feature for subsequent comparison. The image features of each pattern are compared with the pre-constructed target feature. The comparison process may involve calculating the similarity or distance between the two feature sets, depending on the comparison algorithm selected. A similarity threshold is set to determine whether the two feature sets are similar enough to be considered as belonging to the same pattern. When the similarity between the image features of a certain pattern and the target features exceeds the set threshold, the pattern is determined as the first pattern and its position information in the image is recorded. This position information may include the coordinates of the upper left corner, lower right corner, center point, etc. of the pattern, depending on the requirements of the application scenario.

[0092] The target feature is formed by combining the features of the first pixel row (the pixel row where the pixel point appears for the first time) and the second pixel row (the pixel row where the pixel point appears for the last time) of the first pattern. This feature extraction method can accurately capture the boundary information of the first pattern in the third direction, thereby improving the accuracy of feature matching. In the target area, only the image features of all patterns are extracted and compared with the target features. Since the target area is determined based on the historical appearance position information of the first pattern, compared with the comparison in the entire area to be retrieved, this method can significantly reduce the number of patterns that need to be compared, thereby improving the comparison efficiency. By setting a similarity threshold, the corresponding pattern is determined as the first pattern only when the similarity between the image feature and the target feature exceeds the threshold. This strategy can avoid mismatching and ensure the accuracy of positioning. At the same time, recording the position information of the first pattern helps subsequent analysis and processing. The embodiment of the present application is not limited by the size, shape or rotation angle of the first pattern. As long as the first pattern appears in a similar manner in the target area (that is, has similar first and second pixel row features), the embodiment of the present application can accurately locate it. This makes the embodiment of the present application have wide applicability in practical applications.

[0093] Optionally, combining features of the first pixel row and the second pixel row of the first pattern into a target feature comprises:

[0094] Acquire a first feature of the first pixel row of the first pattern, and acquire a second feature of the second pixel row of the first pattern, wherein the first feature includes a first number and a first position of pixels, and the second feature includes a second number and a second position of pixels;

[0095] A first characteristic relationship between the first quantity and the second quantity is established, a second characteristic relationship between the first position and the second position is established, and the first characteristic relationship and the second characteristic relationship are combined into a target characteristic.

[0096] First quantity: In the first pixel row, the number of characteristic pixels belonging to the first pattern is counted. Characteristic pixels are usually identified based on a specific attribute (such as color, brightness, shape, etc.). First position: The specific position of these characteristic pixels in the first pixel row is recorded, usually in the form of column coordinates (or pixel index). Second quantity: In the second pixel row, the number of characteristic pixels belonging to the first pattern is also counted. Second position: The specific position of these characteristic pixels in the second pixel row is recorded, also in the form of column coordinates (or pixel index). The first characteristic relationship refers to a relationship or pattern established based on the characteristic quantity (first quantity) and characteristic position (first position) of the first pixel row. The first characteristic relationship describes the pixel distribution, density or a specific arrangement of the pattern at the top edge. Similarly, the second characteristic relationship is a relationship or pattern established based on the characteristic quantity (second quantity) and characteristic position (second position) of the second pixel row. Combining the first characteristic relationship and the second characteristic relationship forms a target feature. This target feature is a composite feature that integrates the key information of the pattern in the third direction (from the first pixel row to the second pixel row). In practical applications, this target feature can be used for matching and retrieval in an image database or a real-time image to find a pattern similar to the first pattern.

[0097] Specifically, the first number in the first pixel row of the triangle is one, the second number in the second pixel row is multiple, a vertical line perpendicular to the second pixel row is drawn from the pixel point of the first pixel row, the first position is on the vertical line, and the second position is multiple pixel points symmetrically distributed relative to the vertical line. The first number in the first pixel row of the rectangle is multiple, the second number in the second pixel row is multiple, the first number is equal to the second number, and the first position and the second position are the same. The first number in the first pixel row of the straight line is one, the second number is one, and the first position and the second position are the same.

[0098] S140: Correct the first pattern according to the offset.

[0099] The main goal of deskew is to eliminate or reduce the offset of the first pattern in the target image so that it is closer to the ideal position. The ideal position may be determined based on a preset standard, historical data, or other reference information. According to the calculated offset, the first pattern is translated horizontally and / or vertically to move it to the ideal position. If the offset includes a rotation component (i.e., the pattern is rotated in the image), a rotation transformation is required to correct this deviation. In some cases, if the size of the pattern is not as expected (which may be due to factors such as shooting distance, lens focal length, etc.), a scaling transformation may be required to adjust the size of the pattern. For more complex deformation situations, an affine transformation can be used to simultaneously correct multiple deviations such as translation, rotation, and scaling. Specifically, according to the offset and the correction target, the required transformation parameters (such as translation, rotation angle, scaling ratio, etc.) are calculated. Using image processing software or algorithms, the calculated transformation parameters are applied to the first pattern to implement the deskew operation. The image after deskew is verified to ensure that the first pattern has been correctly moved to the ideal position and no new deformation or error has been introduced.

[0100] The method of combining feature extraction with memory method to quickly locate patterns can not only improve the working efficiency of CCD and CMOS in correction, but also the algorithm that imitates the human visual system can be used in various scenarios, such as algorithm improvement of vehicle license plates at parking lot entrances and exits, or classification of objects on logistics conveyor belts, or face recognition systems in schools, etc.

[0101] This embodiment also discloses an image processing system. Figure 5 is a schematic diagram of a module of an image processing system disclosed in an embodiment of the present application, such as Figure 5 As shown, the system includes a collection module 501, a calculation module 502, a positioning module 503 and a correction module 504, wherein:

[0102] The acquisition module 501 is configured to acquire image data through a preset sensor and determine whether a pattern scanning instruction issued by a user is received;

[0103] The calculation module 502 is configured to perform imaging processing on the collected image data when no pattern scanning instruction issued by the user is received, calculate the offset of the first pattern after imaging from the preset center position, and determine whether the preset fast positioning condition is met according to the offset;

[0104] A positioning module 503, configured to determine a target area according to the historical occurrence position information of the first pattern when a preset fast positioning condition is met, and perform feature comparison in the target area to locate the first pattern;

[0105] The correction module 504 is configured to correct the first pattern according to the offset.

[0106] Optionally, the processing system further includes a storage module, and the storage module is configured to:

[0107] When receiving a pattern scanning instruction from the user, display the currently acquired image data and wait for the user to confirm the second pattern to be tracked;

[0108] After the user confirms, the second pattern is feature extracted by the preset sensor, and the extracted feature information of the second pattern is classified and stored.

[0109] Optionally, the calculation module 502 is configured to:

[0110] Acquire characteristic information of the first pattern, and determine a center point of the first pattern and a preset fast positioning condition according to the characteristic information;

[0111] Calculating a first distance and a second distance between the center point and the preset center position, and calculating a first offset between the center point and the preset center position according to the first distance and the second distance, wherein the first distance is a distance in a first direction, and the second distance is a distance in a second direction, and the first direction is perpendicular to the second direction;

[0112] The first offset is compared with an offset threshold in the preset fast positioning condition to determine whether the preset fast positioning condition is met.

[0113] Optionally, the positioning module 503 is configured to:

[0114] Divide the area to be searched into multiple first preset areas according to a first preset rule, and determine whether the frequency of the first pattern appearing alone in any first preset area exceeds a first threshold according to the historical appearance position information, wherein the first preset rule is to draw a straight line perpendicular to the edge according to the equally divided points of all edges of the area to be searched, and divide the area to be searched according to the straight line and the intersection of the straight lines;

[0115] When the frequency of the first pattern appearing alone in the first preset target area exceeds the first threshold, the first preset target area is determined as the target area;

[0116] When the frequency of the first pattern appearing alone in any first preset area does not exceed the first threshold, determining whether the sum of the frequencies of the first pattern appearing in two adjacent first preset areas exceeds a second threshold;

[0117] When the frequency of the first pattern appearing in the second preset target area and the third preset target area exceeds the second threshold, the second preset target area and the third preset target area are determined as target areas, and the second preset target area is adjacent to the third preset target area.

[0118] Optionally, the positioning module 503 is further configured to:

[0119] When the sum of the frequencies of the first pattern appearing in two adjacent first preset areas does not exceed a second threshold, the area to be searched is divided into a plurality of second preset areas according to a second preset rule, wherein the second preset rule is to draw straight lines perpendicular to the target edge according to the equally divided points of the target edge of the area to be searched, and divide the area to be searched according to the straight lines;

[0120] A second preset area where the first pattern appears most frequently is determined as a target area.

[0121] Optionally, the positioning module 503 is configured to:

[0122] combining features of a first pixel row and a second pixel row of the first pattern into a target feature, wherein the first pixel row is a pixel row in which pixels of the first pattern first appear along a third direction, and the second pixel row is a pixel row in which pixels of the first pattern last appear along the third direction;

[0123] Extracting image features of all patterns in the target area, and comparing the image features with the target features;

[0124] A pattern in the target area whose comparison similarity exceeds a similarity threshold is determined as the first pattern, and position information of the first pattern is recorded.

[0125] Optionally, the positioning module 503 is further configured to:

[0126] Acquire a first feature of the first pixel row of the first pattern, and acquire a second feature of the second pixel row of the first pattern, wherein the first feature includes a first number and a first position of pixels, and the second feature includes a second number and a second position of pixels;

[0127] A first characteristic relationship between the first quantity and the second quantity is established, a second characteristic relationship between the first position and the second position is established, and the first characteristic relationship and the second characteristic relationship are combined into a target characteristic.

[0128] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0129] This embodiment also discloses an electronic device, referring to Figure 6 The electronic device may include: at least one processor 601 , at least one communication bus 602 , a user interface 603 , a network interface 604 , and at least one memory 605 .

[0130] The communication bus 602 is used to realize the connection and communication between these components.

[0131] The user interface 603 may include a display screen (Display) and a camera (Camera). The optional user interface 603 may also include a standard wired interface and a wireless interface.

[0132] The network interface 604 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0133] Among them, the processor 601 may include one or more processing cores. The processor 601 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 605, and calling data stored in the memory 605. Optionally, the processor 601 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 601 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 601, and it can be implemented separately through a chip.

[0134] Among them, the memory 605 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 605 includes a non-transitory computer-readable storage medium. The memory 605 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 605 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 605 may optionally be at least one storage device located away from the aforementioned processor 601. As Figure 6 As shown, the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of an image imaging processing method.

[0135] exist Figure 6 In the electronic device shown, the user interface 603 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 601 can be used to call the application program of the image imaging processing method stored in the memory 605. When executed by one or more processors 601, the electronic device executes one or more methods in the above-mentioned embodiments.

[0136] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0137] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 605, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 605 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0142] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for processing image imaging, characterized in that: Applied to an image processing platform, the method comprises: Collect image data through a preset sensor and determine whether a pattern scanning instruction issued by a user is received; When no pattern scanning instruction is received from the user, imaging processing is performed on the collected image data, an offset of the imaged first pattern from a preset center position is calculated, and whether a preset fast positioning condition is met according to the offset; When a preset fast positioning condition is met, determining a target area according to the historical occurrence position information of the first pattern, and performing feature comparison in the target area to locate the first pattern; Correcting the first pattern according to the offset, The calculating the offset of the first pattern after imaging from the preset center position, and determining whether the preset rapid positioning condition is met according to the offset comprises: Acquire characteristic information of the first pattern, and determine a center point of the first pattern and a preset fast positioning condition according to the characteristic information; Calculating a first distance and a second distance between the center point and the preset center position, and calculating a first offset between the center point and the preset center position according to the first distance and the second distance, wherein the first distance is a distance in a first direction, and the second distance is a distance in a second direction, and the first direction is perpendicular to the second direction; comparing the first offset with an offset threshold in the preset fast positioning condition to determine whether the preset fast positioning condition is met, The performing feature comparison in the target area to locate the first pattern includes: combining features of a first pixel row and a second pixel row of the first pattern into a target feature, wherein the first pixel row is a pixel row in which pixels of the first pattern first appear along a third direction, and the second pixel row is a pixel row in which pixels of the first pattern last appear along the third direction; Extracting image features of all patterns in the target area, and comparing the image features with the target features; determining a pattern in the target area whose comparison similarity exceeds a similarity threshold as the first pattern, and recording position information of the first pattern, Combining the features of the first pixel row and the second pixel row of the first pattern into a target feature comprises: Acquire a first feature of the first pixel row of the first pattern, and acquire a second feature of the second pixel row of the first pattern, wherein the first feature includes a first number and a first position of pixels, and the second feature includes a second number and a second position of pixels; A first characteristic relationship between the first quantity and the second quantity is established, a second characteristic relationship between the first position and the second position is established, and the first characteristic relationship and the second characteristic relationship are combined into a target characteristic.

2. The image processing method according to claim 1, characterized in that: The processing method also includes: When receiving a pattern scanning instruction from the user, display the currently acquired image data and wait for the user to confirm the second pattern to be tracked; After the user confirms, the second pattern is feature extracted by the preset sensor, and the extracted feature information of the second pattern is classified and stored.

3. The image processing method according to claim 1, characterized in that: The determining of the target area according to the historical appearance position information of the first pattern comprises: Divide the area to be searched into multiple first preset areas according to a first preset rule, and determine whether the frequency of the first pattern appearing alone in any first preset area exceeds a first threshold according to the historical appearance position information, wherein the first preset rule is to draw a straight line perpendicular to the edge according to the equally divided points of all edges of the area to be searched, and divide the area to be searched according to the straight line and the intersection of the straight lines; When the frequency of the first pattern appearing alone in the first preset target area exceeds the first threshold, the first preset target area is determined as the target area; When the frequency of the first pattern appearing alone in any first preset area does not exceed the first threshold, determining whether the sum of the frequencies of the first pattern appearing in two adjacent first preset areas exceeds a second threshold; When the frequency of the first pattern appearing in the second preset target area and the third preset target area exceeds the second threshold, the second preset target area and the third preset target area are determined as target areas, and the second preset target area is adjacent to the third preset target area.

4. The image processing method according to claim 3, characterized in that: The determining the target area according to the historical appearance position information of the first pattern further comprises: When the sum of the frequencies of the first pattern appearing in two adjacent first preset areas does not exceed a second threshold, the area to be searched is divided into a plurality of second preset areas according to a second preset rule, wherein the second preset rule is to draw straight lines perpendicular to the target edge according to the equally divided points of the target edge of the area to be searched, and divide the area to be searched according to the straight lines; A second preset area where the first pattern appears most frequently is determined as a target area.

5. An image processing system, characterized in that: It includes acquisition module, calculation module, positioning module and correction module, among which: An acquisition module, configured to acquire image data through a preset sensor and determine whether a pattern scanning instruction issued by a user is received; A calculation module, configured to perform imaging processing on the collected image data when no pattern scanning instruction issued by the user is received, calculate the offset of the first pattern after imaging from the preset center position, and determine whether the preset fast positioning condition is met according to the offset; a positioning module configured to determine a target area according to historical occurrence position information of the first pattern when a preset fast positioning condition is met, and perform feature comparison in the target area to locate the first pattern; a correction module, configured to correct the first pattern according to the offset, The calculating the offset of the first pattern after imaging from the preset center position, and determining whether the preset rapid positioning condition is met according to the offset comprises: Acquire characteristic information of the first pattern, and determine a center point of the first pattern and a preset fast positioning condition according to the characteristic information; Calculating a first distance and a second distance between the center point and the preset center position, and calculating a first offset between the center point and the preset center position according to the first distance and the second distance, wherein the first distance is a distance in a first direction, and the second distance is a distance in a second direction, and the first direction is perpendicular to the second direction; comparing the first offset with an offset threshold in the preset fast positioning condition to determine whether the preset fast positioning condition is met, The performing feature comparison in the target area to locate the first pattern includes: combining features of a first pixel row and a second pixel row of the first pattern into a target feature, wherein the first pixel row is a pixel row in which pixels of the first pattern first appear along a third direction, and the second pixel row is a pixel row in which pixels of the first pattern last appear along the third direction; Extracting image features of all patterns in the target area, and comparing the image features with the target features; determining a pattern in the target area whose comparison similarity exceeds a similarity threshold as the first pattern, and recording position information of the first pattern, Combining the features of the first pixel row and the second pixel row of the first pattern into a target feature comprises: Acquire a first feature of the first pixel row of the first pattern, and acquire a second feature of the second pixel row of the first pattern, wherein the first feature includes a first number and a first position of pixels, and the second feature includes a second number and a second position of pixels; A first characteristic relationship between the first quantity and the second quantity is established, a second characteristic relationship between the first position and the second position is established, and the first characteristic relationship and the second characteristic relationship are combined into a target characteristic.

6. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 4 is performed.

Images