Image Processing Method and Apparatus, Storage Medium, and Electronic Device

Abstract

The present invention discloses an image processing method, apparatus, storage medium and electronic device. Among them, the above method includes: obtaining a difference region in a first image and a second image; wherein, the first image and the second image are images to be compared; respectively determining a minimum bounding rectangle of the contour corresponding to each difference region to obtain a set of minimum bounding rectangles; merging multiple bounding rectangles in the set of minimum bounding rectangles whose adjacency or overlap degree is greater than or equal to a preset threshold to obtain a target set of bounding rectangles; the adjacency or overlap degree is the ratio of a first area to a second area; wherein, the first area is the sum of the areas of any two bounding rectangles in the set of minimum bounding rectangles, and the second area is the area of the minimum bounding rectangle of the union of the any two bounding rectangles. The present invention solves the technical problem of low efficiency in extracting image change information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly, to an image processing method, apparatus, storage medium, and electronic device. Background Art

[0002] Automatic detection of image change regions is an important means for effectively extracting image change information and is the basis for utilizing image change information. Currently, there are many studies focusing on the detection of image change regions, such as image change detection based on Scale-Invariant Feature Transform (SIFT) feature points, or using means such as pattern recognition for comparative analysis, etc. However, in related technologies for image change detection, only the image information contained in the identified multiple change regions is extracted separately, and the adjacent change regions of the image are not effectively integrated. Therefore, the efficiency of extracting information from the image change regions is relatively low. Summary of the Invention

[0003] Embodiments of the present invention provide an image processing method, apparatus, storage medium, and electronic device to at least solve the technical problem of relatively low efficiency in extracting image change information.

[0004] According to one aspect of the embodiments of the present invention, there is provided an image processing method, including: obtaining a difference region between a first image and a second image; wherein, the first image and the second image are images to be compared; respectively determining the minimum circumscribed rectangle of the contour corresponding to each difference region to obtain a set of minimum circumscribed rectangles; merging multiple circumscribed rectangles in the set of minimum circumscribed rectangles whose adjacency or overlap degree is greater than or equal to a preset threshold to obtain a target set of circumscribed rectangles; wherein, the adjacency or overlap degree is the ratio of a first area to a second area; the first area is the sum of the areas of any two circumscribed rectangles in the set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the any two circumscribed rectangles.

[0005] According to another aspect of the embodiments of the present invention, there is also provided an image processing apparatus, including: an acquisition unit configured to acquire a difference region between a first image and a second image, where the first image and the second image are images to be compared; a first determination unit configured to respectively determine a minimum circumscribed rectangle of the contour corresponding to each difference region to obtain a set of minimum circumscribed rectangles; a merging unit configured to merge multiple circumscribed rectangles in the set of minimum circumscribed rectangles whose adjacent or overlapping degree is greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles, where the adjacent or overlapping degree is a ratio of a first area to a second area, the first area is the sum of the areas of any two circumscribed rectangles in the set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the any two circumscribed rectangles.

[0006] According to still another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to execute the above-mentioned image processing method through the computer program.

[0007] According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium in which a computer program is stored, where the computer program is configured to execute the above-mentioned image processing method when running.

[0008] In the embodiments of the present invention, a method is adopted, including: acquiring a difference region between a first image and a second image, where the first image and the second image are images to be compared; respectively determining a minimum circumscribed rectangle of the contour corresponding to each difference region to obtain a set of minimum circumscribed rectangles; merging multiple circumscribed rectangles in the set of minimum circumscribed rectangles whose adjacent or overlapping degree is greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles, where the adjacent or overlapping degree is a ratio of a first area to a second area, the first area is the sum of the areas of any two circumscribed rectangles in the set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the any two circumscribed rectangles.

[0009] In the above method, since adjacent difference regions of two images to be compared are merged, to effectively merge the difference regions, multiple circumscribed rectangles in the set of minimum circumscribed rectangles with an adjacent or overlapping degree greater than or equal to a preset threshold are selected for merging. And the above adjacent or overlapping degree is the ratio of a first area to a second area. The first area is the sum of the areas of any two circumscribed rectangles in the set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the above any two circumscribed rectangles. In this way, not only the number of difference regions is reduced, the repeated extraction of information is avoided, the technical effects of improving the extraction efficiency of image change information and reducing the recognition information calculation resources of the system are achieved, and furthermore, the technical problem of low extraction efficiency of image change information is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0011] Figure 1 is a schematic diagram of an application environment of an optional image processing method according to an embodiment of the present invention;

[0012] Figure 2 is a schematic diagram of an application environment of another optional image processing method according to an embodiment of the present invention;

[0013] Figure 3 is a schematic flowchart of an optional image processing method according to an embodiment of the present invention;

[0014] Figure 4 is a schematic diagram of an image to be compared in an optional image processing method according to an embodiment of the present invention;

[0015] Figure 5 is a schematic diagram of another image to be compared in an optional image processing method according to an embodiment of the present invention;

[0016] Figure 6 is a schematic diagram of the difference in the change regions of two images to be compared according to an embodiment of the present invention;

[0017] Figure 7 is a schematic diagram of the binarization of the change regions of two images to be compared according to an embodiment of the present invention;

[0018] Figure 8 is a schematic diagram of the contour of the change region of an optional image processing method according to an embodiment of the present invention;

[0019] Figure 9Schematic diagram of the circumscribed rectangle of the changed area of an optional image processing method according to an embodiment of the present invention;

[0020] Figure 10 Schematic diagram of the calculation of the intersection over union of the circumscribed rectangles of an optional image processing method according to an embodiment of the present invention;

[0021] Figure 11 Schematic diagram of the integrated display of the changed area of an optional image processing method according to an embodiment of the present invention;

[0022] Figure 12 Schematic diagram of the changed area detection process according to an embodiment of the present invention;

[0023] Figure 13 Schematic diagram of the changed area merging process according to an embodiment of the present invention;

[0024] Figure 14 Schematic diagram of the structure of an optional image processing apparatus according to an embodiment of the present invention;

[0025] Figure 15 Schematic diagram of the structure of an optional electronic device according to an embodiment of the present invention. Detailed implementation manners

[0026] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0028] According to one aspect of the embodiments of the present invention, an image processing method is provided. Optionally, as an alternative implementation, the above image processing method can be but is not limited to being applied to an application environment as shown in Figure 1 . The application environment includes: a terminal device 102 for human-computer interaction with the user, a network 104, and a server 106. Human-computer interaction can be carried out between the user 108 and the terminal device 102, and an image processing application program runs in the terminal device 102. The above terminal device 102 includes a human-computer interaction screen 1022, a processor 1024, and a memory 1026. The human-computer interaction screen 1022 is used to present a first image and a second image; the processor 1024 is used to obtain the first image and the second image to be compared, send an image processing request, and receive a set of target circumscribed rectangles sent by the server 106. The memory 1026 is used to store the above set of target circumscribed rectangles.

[0029] In addition, the server 106 includes a database 1062 and a processing engine 1064. The database 1062 is used to store the above target advertisement and the coding identifier of the target advertisement, and is used to store the first image, the second image to be compared, and the set of target circumscribed rectangles. The processing engine 1064 is used to obtain the difference regions in the first image and the second image; wherein, the above first image and second image are images to be compared; respectively determine the minimum circumscribed rectangles of the contours corresponding to each difference region to obtain a set of minimum circumscribed rectangles; merge multiple circumscribed rectangles in the above set of minimum circumscribed rectangles whose adjacency or overlap degree is greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles; send the set of target circumscribed rectangles to the client of the above terminal device 102.

[0030] In one or more embodiments, the above image processing method of the present application can be applied to an application environment as shown in Figure 2 . As shown in Figure 2 , human-computer interaction can be carried out between the user 202 and the user device 204. The user device 204 includes a memory 206 and a processor 208. In this embodiment, the user device 204 can be but is not limited to performing the operations performed by the above terminal device 102 to obtain a set of target circumscribed rectangles.

[0031] Optionally, the above-mentioned terminal device 102 and user device 204 include, but are not limited to, terminals such as mobile phones, tablet computers, laptop computers, PCs, in-vehicle electronic devices, and wearable devices. The above-mentioned network 104 may include, but is not limited to, a wireless network or a wired network. Among them, the wireless network includes: WIFI and other networks that implement wireless communication. The above-mentioned wired network may include, but is not limited to: wide area network, metropolitan area network, local area network. The above-mentioned server 106 may include, but is not limited to, any hardware device that can perform calculations. The above-mentioned server may be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example, and this embodiment does not make any limitation thereto.

[0032] Optionally, in this embodiment, the above-mentioned terminal device 102 may be a terminal device configured with an image processing client, and may include, but is not limited to, at least one of the following: mobile phone (such as Android mobile phone, iOS mobile phone, etc.), laptop computer, tablet computer, handheld computer, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, etc. The target client may be a video client, instant messaging client, browser client, education client, etc. The above-mentioned network 104 may include, but is not limited to: wired network, wireless network. Among them, the wired network includes: local area network, metropolitan area network, and wide area network. The wireless network includes: Bluetooth, WIFI, and other networks that implement wireless communication. The above-mentioned server 106 may be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example, and this embodiment does not make any limitation thereto.

[0033] As an optional implementation manner, as Figure 3 shown, the embodiment of the present invention provides an image processing method, including the following steps:

[0034] S302, obtain the difference region between the first image and the second image; wherein, the above-mentioned first image and second image are images to be compared.

[0035] In the embodiment of the present invention, the first image and the second image include, but are not limited to, two temporally adjacent frames of images obtained from a video; they may also be two images intercepted from a web page or a client. Here, the embodiment of the present invention includes, but is not limited to, obtaining the difference region between the first image and the second image by means of image change detection based on SIFT feature points, or using means such as pattern recognition for comparative analysis, or change detection based on structural similarity, etc. No limitation is made here.

[0036] S304, respectively determine the minimum circumscribed rectangle of the contour corresponding to each difference region to obtain a set of minimum circumscribed rectangles.

[0037] Specifically, in the embodiments of the present invention, a contour search method is used to search for the contours corresponding to the difference regions between the first image and the second image, and the minimum bounding rectangles corresponding to the contours are obtained, so as to obtain a set of minimum bounding rectangles formed by combining the minimum bounding rectangles corresponding to the contours.

[0038] S306. Merge multiple bounding rectangles in the set of minimum bounding rectangles whose adjacency or overlap degree is greater than or equal to a preset threshold to obtain a set of target bounding rectangles; the adjacency or overlap degree is the ratio of the first area to the second area; wherein, the first area is the sum of the areas of any two bounding rectangles in the set of minimum bounding rectangles, and the second area is the area of the minimum bounding rectangle of the union of the any two bounding rectangles.

[0039] Here, in the embodiments of the present invention, the Bounding Intersection over Union (BIoU) is used as the adjacency or overlap degree between two bounding rectangles. The definition of the BIoU is the ratio of the sum of the areas of two rectangles to the area of the minimum bounding rectangle of the union of the two rectangles, which can measure the adjacency or overlap degree between two rectangles. If the overlap degree of two rectangles is higher or closer, the BIoU value is larger, otherwise the BIoU value is smaller. Its formula is:

[0040]

[0041] where the Area() function obtains the area of the input rectangle parameter, the bounding rectangle() function obtains the bounding rectangle of the input shape parameter, and c i and c j are two rectangles to be compared.

[0042] As Figure 10 shown, Figure 10 in (A) is the case where there is an intersection between the two rectangles c i and c j , (B) is the case when the two rectangles c i and c j are close, (C) is the case when there is an inclusion relationship between the two rectangles c i and c j , where the bounding rectangles abcd are the minimum bounding rectangles of the union of the two rectangles c i and c j in the above three cases respectively.

[0043] In an embodiment of the present invention, for example, when two circumscribed rectangles are adjacent but have no intersection, the above-mentioned degree of adjacency or overlap includes, but is not limited to, the distance between the adjacent sides of the two adjacent circumscribed rectangles. That is, the smaller the distance between the adjacent sides of the two adjacent circumscribed rectangles, the greater the degree of adjacency or overlap. When two circumscribed rectangles have an inclusion relationship, the degree of adjacency or overlap between them is a state greater than a preset threshold; when two circumscribed rectangles have an intersection, the larger the intersection between them, the greater the degree of adjacency or overlap. At this time, after merging two circumscribed rectangles with a degree of adjacency or overlap greater than or equal to the preset threshold, the minimum circumscribed rectangle of the union of the two circumscribed rectangles will be obtained. Replace one of the two circumscribed rectangles with the minimum circumscribed rectangle, and then delete the other circumscribed rectangle; or directly delete the two merged circumscribed rectangles, retain the minimum circumscribed rectangle of the two circumscribed rectangles, and continue to determine the degree of adjacency or overlap of the minimum circumscribed rectangle with other circumscribed rectangles, and determine whether to merge according to the degree of adjacency or overlap until the degree of adjacency or overlap between every two circumscribed rectangles in the above-mentioned minimum circumscribed rectangle set is less than the preset threshold, and then determine to obtain the above-mentioned target circumscribed rectangle set.

[0044] In an embodiment of the present invention, a method is adopted to obtain the difference regions in the first image and the second image; wherein, the above-mentioned first image and the second image are images to be compared; respectively determine the minimum circumscribed rectangles of the contours corresponding to each difference region to obtain a set of minimum circumscribed rectangles; and merge multiple circumscribed rectangles in the above-mentioned set of minimum circumscribed rectangles with a degree of adjacency or overlap greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles.

[0045] In the above method, to effectively merge the difference regions, the present invention uses the ratio of the first area to the second area to measure the adjacent or overlapping degree of any two rectangles. In the related art, the method for calculating the overlapping degree of any two rectangles is the Intersection over Union (IoU), where IoU is the ratio of the intersection of any two rectangles to the union of the any two rectangles. In practice, when two rectangles are closely adjacent but do not intersect, the corresponding IoU value will be equal to 0; or when two rectangles have an inclusion relationship and one rectangle is much larger than the other rectangle, the corresponding IoU value will approach 0. This results in the inability to merge two rectangles with a large adjacent or overlapping degree in the above two cases. Therefore, the present invention uses the BIoU ratio of the sum of the areas of any two circumscribed rectangles to the area of the minimum circumscribed rectangle of the union of the any two circumscribed rectangles to measure the adjacent or overlapping degree of any two rectangle frames. When two rectangles are closely adjacent but do not intersect, the corresponding BIoU value will approach 1; or when two rectangles have an inclusion relationship and one rectangle is much larger than the other rectangle, the corresponding BIoU value will approach 1. This makes the adjacent or overlapping degree of any two rectangle frames positively correlated with the BIoU ratio adopted by the present invention, enabling the merger of two rectangles with a high adjacent or overlapping degree. In the actual processing of image change information, the BIoU ratio proposed in the embodiments of the present invention can reasonably and effectively measure whether to merge the difference regions, reduce the invalid difference regions, and avoid the repeated extraction of information. At the same time, it merges the information that should be merged, avoiding the problem of being extracted as information within two difference regions and then consuming computing resources by going through a cumbersome subsequent process to determine whether to merge.

[0046] In the above method, since the adjacent difference regions of the two images to be compared are merged, the number of difference regions is reduced, achieving the technical effects of improving the efficiency of image change information extraction and reducing the computing resources for system recognition information, thereby solving the technical problem of low efficiency in image change information extraction.

[0047] In one or more embodiments, multiple circumscribed rectangles in the above minimum circumscribed rectangle set with an adjacent or overlapping degree greater than or equal to a preset threshold are merged to obtain a target circumscribed rectangle set, including:

[0048] Taking any circumscribed rectangle in the above minimum circumscribed rectangle set as the current circumscribed rectangle, and performing the following operations on the current circumscribed rectangle until all circumscribed rectangles are traversed:

[0049] Comparing the adjacent or overlapping degree between the above current circumscribed rectangle and another circumscribed rectangle outside the above current circumscribed rectangle;

[0050] When the above adjacent or overlapping degree is greater than or equal to the preset threshold, replace the current circumscribed rectangle with the minimum circumscribed rectangle of the union of the current circumscribed rectangle and the other circumscribed rectangle, and delete the other circumscribed rectangle from the set of minimum circumscribed rectangles;

[0051] When the adjacent or overlapping degree between every two circumscribed rectangles in the set of minimum circumscribed rectangles is less than the preset threshold, the target set of circumscribed rectangles is determined.

[0052] Specifically, in the process of effectively merging multiple change regions detected in the first image and the second image. Among the multiple change regions, there are cases where some change regions contain one or more change regions, or there are cases where some change regions have a high overlap degree with one or more change regions, or there are cases where some change regions are closely adjacent to one or more other change regions. To effectively extract the change regions of the image, it is necessary to merge regions with a high overlap degree, or an inclusion relationship, or close adjacency.

[0053] Here, if the number of elements in the set of minimum circumscribed rectangles is 1, there is no need to merge regions, and the final set of image change regions is directly obtained.

[0054] If the number of elements in the set of minimum circumscribed rectangles is greater than 1, it is necessary to determine whether to merge the elements in the set of minimum circumscribed rectangles. The process is as follows:

[0055] For each circumscribed rectangle in the set of circumscribed rectangles, traverse from the first circumscribed rectangle to the last circumscribed rectangle, and perform the following operations:

[0056] Record the first circumscribed rectangle being traversed as the actively merged circumscribed rectangle, and the circumscribed rectangles other than the first circumscribed rectangle as the passively merged circumscribed rectangles. The merging process needs to calculate the value of the adjacent or overlapping degree between the actively merged circumscribed rectangle and the passively merged circumscribed rectangles. If the value of the adjacent or overlapping degree is greater than or equal to the preset threshold, that is, the merging condition is met, then the actively merged circumscribed rectangle and the passively merged circumscribed rectangle need to be merged to obtain a new circumscribed rectangle. In the set of circumscribed rectangles, replace the actively merged circumscribed rectangle in the set of minimum circumscribed rectangles with the new circumscribed rectangle. At the same time, delete the passively merged circumscribed rectangle from the set of minimum circumscribed rectangles, and all the subsequent passively merged circumscribed rectangles are moved forward by one position. Continue to calculate the value of the adjacent or overlapping degree between the actively merged circumscribed rectangle and the subsequent passively merged circumscribed rectangles.

[0057] Among them, the merging method of the actively merged circumscribed rectangle and the passively merged circumscribed rectangle is: take the union of the two circumscribed rectangles, and then obtain the minimum circumscribed rectangle of the union.

[0058] If the value of the adjacent or overlapping degree is less than the preset threshold, the passive merged bounding rectangle is retained, and the value of the adjacent or overlapping degree between the active merged bounding rectangle and the next passive merged bounding rectangle is continuously calculated until the values of the adjacent or overlapping degrees between the active merged bounding rectangle and the subsequent passive merged bounding rectangles are all calculated, and until the last bounding rectangle is traversed.

[0059] After the traversal is completed, the final set of changed regions, that is, the set of target bounding rectangles, is obtained.

[0060] In one or more embodiments, the above image processing method further includes: sorting each bounding rectangle in the above minimum bounding rectangle set in descending order according to the area of each bounding rectangle; determining the order of the current bounding rectangle according to the result of the above descending order.

[0061] In the embodiments of the present invention, by sorting the areas of each bounding rectangle in the minimum bounding rectangle set from large to small, and first using the bounding rectangle with a larger area as the current rectangle to judge the adjacent or overlapping degree with other rectangles, not only can the bounding rectangles with smaller areas be avoided from being missed during the merging of bounding rectangles, but also the efficiency of extracting image change information can be improved.

[0062] In one or more embodiments, the above image processing method further includes: when the adjacent or overlapping degree between the current bounding rectangle and the other bounding rectangle is less than the preset threshold, continue to compare the current bounding rectangle with the remaining bounding rectangles in the above minimum bounding rectangle set.

[0063] In one or more embodiments, after obtaining the above minimum bounding rectangle set, it further includes:

[0064] Determine the above adjacent or overlapping degree according to the ratio of the above first area to the above second area.

[0065] Here, the embodiments of the present invention use the Bounding Intersection over Union (BIoU) as the adjacent or overlapping degree between two bounding rectangles. It is the ratio of the sum of the areas of two rectangles to the area of the minimum bounding rectangle of the union of the two rectangles, and can measure the overlap degree of the two rectangles. If the overlap degree of the two rectangles is higher or closer, the BIoU value is larger, otherwise the BIoU value is smaller. Its formula is:

[0066]

[0067] Among them, the Area() function obtains the area of the input rectangle parameter, and the bounding rectangle() function obtains the bounding rectangle of the input shape parameter, c i and c jTwo rectangles to be compared.

[0068] As Figure 10 shown, Figure 10 in (A) is the case where there is an intersection between rectangle c i and rectangle c j ; (B) is the case where rectangle c i and rectangle c j are close; (C) is the case where there is an inclusion relationship between rectangle c i and rectangle c j . Among them, the circumscribed rectangle abcd is the minimum circumscribed rectangle of the union of rectangle c i and rectangle c j in the above three cases.

[0069] In one or more embodiments, obtaining the difference regions in the first image and the second image as described above includes:

[0070] Performing grayscale processing on the first image and the second image;

[0071] Determining the pixel points at the same positions in the grayscale-processed first image and second image, and calculating the structural similarity of each of the pixel points at the same positions;

[0072] Determining the difference regions based on the above structural similarity.

[0073] In an embodiment of the present invention, taking the image shown in Figure 4 as the first image and the image shown in Figure 5 as the second image as an example, the first image and the second image have the same size and the same resolution. The first image and the second image are subjected to grayscale processing to generate the corresponding grayscale images of the first image and the second image. Then, based on the structural similarity SSIM algorithm, the structural similarity of each pixel point at the same position in the two grayscale images is calculated, and finally, a difference effect image including the difference regions of the two pictures as shown in Figure 6 is obtained.

[0074] In one or more embodiments, respectively determining the minimum circumscribed rectangles of the contours corresponding to each difference region to obtain a set of minimum circumscribed rectangles includes:

[0075] Performing binarization processing on each of the above difference regions;

[0076] Determining the contours of the binarized difference regions based on a preset contour search algorithm to obtain a set of difference contours;

[0077] Taking the minimum circumscribed rectangles of all the parent contours in the set of difference contours as the set of minimum circumscribed rectangles.

[0078] In an embodiment of the present invention, it includes, but is not limited to, performing binarization processing on the difference effect diagram in combination with the Otsu method to obtain a binarized image as shown in Figure 7 . Then, use the contour finding method to find the contours of the differences between the two pictures on the binarized image as shown in Figure 7 to obtain a set of difference contours P, and an image of the set of difference contours P including the first image and the second image as shown in Figure 8 , where the contours are presented as dots.

[0079] Traverse the set of difference contours P and determine whether it is a parent contour. If so, add it to the set of parent contours. Finally, obtain the set of parent contours Q. Traverse the set of parent contours Q, calculate the areas of all parent contours, and obtain the corresponding set of parent contour areas A = {A a , A b ,..., A m}, m ∈ N + , where m is the number of elements in the set.

[0080] Then perform a descending order operation on the set of parent contour areas A. For each contour, find its bounding rectangle. Obtain the set of bounding rectangles of all contours C = {C a , C b ,..., C m}. Use C x = [x 1 , y 1 , x 2 , y 2 to represent the rectangle, where (x 1 , y 1 ) and (x 2 , y 2 ) are the coordinates of the lower left corner and the upper right corner of the rectangle respectively. The upper and lower sides of the rectangle are parallel to the upper and lower sides of the current display image, and the left and right sides are parallel to the left and right sides of the current display image. As shown in Figure 9 , Figure 9 shows multiple changing regions of the first image and the second image with rectangles. Merge multiple bounding rectangles in the above-mentioned set of minimum bounding rectangles (the set of bounding rectangles of all contours C) whose adjacent or overlapping degree is greater than or equal to a preset threshold to obtain an image including the set of target bounding rectangles as shown in Figure 11 .

[0081] In one or more embodiments, after obtaining the set of target bounding rectangles, the above image processing method further includes:

[0082] Identifying the image information included in each bounding rectangle in the set of target bounding rectangles; outputting the above image information.

[0083] In an embodiment of the present invention, by recognizing the image information contained in each circumscribed rectangle in the set of target circumscribed rectangles such as Figure 11 For example, the characters in each circumscribed rectangle are recognized by Optical Character Recognition (OCR), and then the above image information (characters) is output.

[0084] In an embodiment of the present invention, since the adjacent difference regions of the two images to be compared are merged, the number of difference regions is reduced, achieving the technical effects of improving the efficiency of extracting image change information and reducing the computing resources of system recognition information, thereby solving the technical problem of low efficiency in extracting image change information.

[0085] Based on the above embodiment, in an application embodiment, the above image processing method further includes:

[0086] S1. Obtain the image change region based on the structural similarity. The difference between two images is obtained by using the Structural Similarity Index Measure (SSIM) algorithm, and the contours are found on the basis of binarization, and finally the circumscribed rectangles of one or more change regions are obtained.

[0087] As Figure 12 shown, the above step S1 specifically includes the following steps:

[0088] 1. Obtain two images to be compared ( Figure 4 and Figure 5 ), Figure 4 is the image before change, Figure 5 is the image after change, the sizes of the two images are the same, and the image resolutions are the same.

[0089] 2. Grayscale the above two images respectively to generate grayscale images to be compared.

[0090] 3. Calculate the structural similarity of the two grayscale images based on the structural similarity and obtain the difference effect diagram of the two pictures ( Figure 6 ).

[0091] 4. Combine the Otsu method to binarize the above difference effect diagram to obtain the binarization effect diagram as Figure 7 shown.

[0092] 5. Use the contour search method to find the contours of the differences between the two pictures on the binarized image, obtain the difference contour set P, and obtain the contour map containing P ( Figure 8 ), where the contours are presented in dots.

[0093] 6. Traverse the difference contours in the difference contour set P, judge whether they are parent contours, if so, add them to the parent contour set, and finally obtain the parent contour set Q.

[0094] 7. Traverse the parent contour set, calculate the area of ​​all parent contours, and obtain the corresponding parent contour area set A = {A a ,A b ,...,A m},m∈N + , the number of elements in the set is m.

[0095] 8. Perform descending operation on the parent contour area set A.

[0096] 9. For each contour, find its circumscribed rectangle. Get the set of circumscribed rectangles of all contours C = {C a ,C b ,...,C m}. Using C x =[x 1 ,y 1 ,x 2 ,y 2 ] to represent a rectangle, where (x 1 ,y 1 ) and (x 2 ,y 2 ) are the coordinates of the lower left corner and upper right corner of the rectangle respectively. The upper and lower sides of the rectangle are parallel to the upper and lower sides of the current displayed image, and the left and right sides are parallel to the left and right sides of the current displayed image. Figure 9 As shown, Figure 9 The plurality of adjacent or overlapping bounding rectangles in the above minimum bounding rectangle set (the set C of bounding rectangles of all contours) that are greater than or equal to a preset threshold are merged to obtain the following: Figure 11 The image shown contains the set of bounding rectangles of the targets.

[0097] S2, determine the degree of overlap BIoU between two rectangles;

[0098] This step mainly gives the calculation method of BIoU value; the technical method of BIoU value is as follows:

[0099]

[0100] Among them, the Area() function obtains the area of ​​the input rectangle parameter, and the bounding rectangle() function obtains the circumscribed rectangle of the input shape parameter. i and c j The two rectangles to be compared.

[0101] S3. Merge the changed regions based on the BIoU.

[0102] This step mainly performs effective merging on the m changed regions detected in step S1. Among the m changed regions, there are cases where one or more changed regions are contained within some changed regions, or cases where some changed regions have a high overlap with one or more changed regions, or cases where some changed regions are closely adjacent to one or more other changed regions. To effectively extract the changed regions of the image, it is necessary to merge regions with high overlap, or with an inclusion relationship, or that are closely adjacent. As Figure 13 shown, this step specifically includes the following content:

[0103] 1. If the number of elements m in the set C of bounding rectangles is 1, then there is no need to merge regions, and the final merged changed region set C = {C 1} is directly output. If the number of elements m in the set C of bounding rectangles is > 1, then it is necessary to merge the changed regions, and the process is as follows:

[0104] 2. Traverse the elements in the set C of bounding rectangles, and denote the currently traversed bounding rectangle as the active merging rectangle c i , and the next bounding rectangle after c i as the passive merging rectangle c j .

[0105] 3. Calculate the BIoU value between c i and c j , and determine whether it is greater than or equal to the threshold.

[0106] 4. If the BIoU value of the two is greater than or equal to the threshold:

[0107] (1) Then merge c i and c j , and after merging, a new bounding rectangle c i' is obtained. In the set C, use c i' to replace c j , and delete c j .

[0108] (2) Determine whether the BIoU values of c i and all its subsequent passive merging rectangles have been calculated.

[0109] ① If not, then continue to calculate the BIoU value between c i and the new passive merging rectangle c j .

[0110] ② If so, then determine whether c i is the penultimate element in the set C.

[0111] 1) If so, then the final merged region set C is obtained.

[0112] 2) If not, start traversing the new active merging rectangle c i , that is, i = i + 1, j = i + 1.

[0113] 5. If the BIoU value of the two is less than the threshold:

[0114] (1) Then retain the circumscribed rectangle c of the passive merging rectangle j .

[0115] (2) Determine whether c j is the last element in set C.

[0116] ① If not, it is necessary to continue calculating the BIoU value of this active merging rectangle c i and the next passive merging rectangle c j , that is, j = j + 1.

[0117] ② If so, determine whether c i is the penultimate element in set C.

[0118] 1) If so, obtain the final merged region set C.

[0119] 2) If not, start traversing the new active merging rectangle c i , that is, i = i + 1, j = i + 1.

[0120] 6. Among them, the merging method of the active merging circumscribed rectangle c i and the passive merging circumscribed rectangle c j involved in steps 4 and 5 is: take the union of the two circumscribed rectangles, and then obtain the circumscribed rectangle c i ' of the union. The circumscribed rectangle c i ' is the merging result of the circumscribed rectangle c i and c j .

[0121] 7. After the traversal ends, obtain the final changed region set C’, such as Figure 11 the image of the target circumscribed rectangle set (changed region set C’) shown.

[0122] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0123] According to another aspect of the embodiments of the present invention, there is also provided an image processing apparatus for implementing the above image processing method. As Figure 14 shown, the apparatus includes:

[0124] An acquisition unit 1402, configured to acquire a difference region in a first image and a second image; wherein, the first image and the second image are images to be compared;

[0125] A first determination unit 1404, configured to respectively determine a minimum circumscribed rectangle of the contour corresponding to each difference region, and obtain a set of minimum circumscribed rectangles;

[0126] A merging unit 1406, configured to merge multiple circumscribed rectangles in the set of minimum circumscribed rectangles that are adjacent or have an overlapping degree greater than or equal to a preset threshold, and obtain a set of target circumscribed rectangles.

[0127] In the embodiments of the present invention, since adjacent difference regions of two images to be compared are merged, in order to effectively merge the difference regions, multiple circumscribed rectangles in the set of minimum circumscribed rectangles that are adjacent or have an overlapping degree greater than or equal to a preset threshold are selected for merging, and the adjacent or overlapping degree is the ratio of a first area to a second area. The first area is the sum of the areas of any two circumscribed rectangles in the set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the any two circumscribed rectangles. In this way, not only the number of difference regions is reduced, the repeated extraction of information is avoided, the technical effect of improving the extraction efficiency of image change information and reducing the computing resources of system recognition information is achieved, and furthermore, the technical problem of low extraction efficiency of image change information is solved.

[0128] In one or more embodiments, the above merging unit 1406 specifically includes:

[0129] An execution module, configured to use any circumscribed rectangle in the set of minimum circumscribed rectangles as the current circumscribed rectangle, and perform the following operations on the current circumscribed rectangle until all circumscribed rectangles are traversed:

[0130] A first comparison module, configured to compare the adjacent or overlapping degree between the current circumscribed rectangle and another circumscribed rectangle outside the current circumscribed rectangle;

[0131] A replacement module, configured to, when the adjacent or overlapping degree is greater than or equal to a preset threshold, replace the current circumscribed rectangle with the minimum circumscribed rectangle of the current circumscribed rectangle and the another circumscribed rectangle, and delete the another circumscribed rectangle from the set of minimum circumscribed rectangles;

[0132] A first determination module, configured to determine the set of target circumscribed rectangles when the adjacent or overlapping degree between every two circumscribed rectangles in the set of minimum circumscribed rectangles is less than a preset threshold.

[0133] In one or more embodiments, the above-mentioned message processing device further includes:

[0134] A sorting unit, configured to sort each circumscribed rectangle in descending order according to the area of each circumscribed rectangle in the above-mentioned minimum circumscribed rectangle set;

[0135] An order determination unit, configured to determine the order of the current circumscribed rectangle according to the result of the above-mentioned descending order.

[0136] In one or more embodiments, the above-mentioned merging unit 1406 further includes:

[0137] A second comparison module, configured to continue comparing the current circumscribed rectangle with the remaining circumscribed rectangles in the above-mentioned minimum circumscribed rectangle set when the degree of adjacency or overlap between the current circumscribed rectangle and the other circumscribed rectangle is less than a preset threshold.

[0138] In one or more embodiments, the above-mentioned message processing device further includes:

[0139] A second determination unit, configured to determine the degree of adjacency or overlap according to the ratio of the above-mentioned one area to the second area.

[0140] In one or more embodiments, the above-mentioned acquisition unit 1402 specifically includes:

[0141] A first processing module, configured to perform grayscale processing on the above-mentioned first image and second image;

[0142] A second determination module, configured to determine pixel points at the same positions in the first image and the second image after grayscale processing, and calculate the structural similarity of each of the above-mentioned pixel points at the same positions;

[0143] A third determination module, configured to determine the above-mentioned difference region based on the above-mentioned structural similarity.

[0144] In one or more embodiments, the above-mentioned acquisition unit 1406 specifically includes:

[0145] A second processing module, configured to perform binarization processing on each of the above-mentioned difference regions;

[0146] A fourth determination module, configured to determine the contour of the above-mentioned difference region after binarization processing based on a preset contour search algorithm to obtain a set of difference contours;

[0147] A fifth determination module, configured to use the minimum circumscribed rectangle of all the parent contours in the above-mentioned set of difference contours as the above-mentioned minimum circumscribed rectangle set.

[0148] In one or more embodiments, the above-mentioned message processing device further includes:

[0149] An identification unit for identifying the image information included in each circumscribed rectangle in the above-mentioned set of target circumscribed rectangles;

[0150] An output unit for outputting the above-mentioned image information.

[0151] According to another aspect of the embodiments of the present invention, there is also provided an electronic device for implementing the above-mentioned image processing method. The electronic device may be Figure 15 the terminal device or server shown. In this embodiment, the electronic device is taken as an example of a terminal for illustration. As Figure 15 shown, the electronic device includes a memory 1502 and a processor 1504. A computer program is stored in the memory 1502, and the processor 1504 is configured to execute the steps in any one of the above-mentioned method embodiments through the computer program.

[0152] Optionally, in this embodiment, the above-mentioned electronic device may be at least one network device among multiple network devices in a computer network.

[0153] Optionally, in this embodiment, the above-mentioned processor may be configured to execute the following steps through a computer program:

[0154] S1. Obtain the difference region between the first image and the second image; wherein, the above-mentioned first image and second image are images to be compared;

[0155] S2. Respectively determine the minimum circumscribed rectangle of the contour corresponding to each difference region to obtain a set of minimum circumscribed rectangles;

[0156] S3. Merge multiple circumscribed rectangles in the above-mentioned set of minimum circumscribed rectangles whose adjacency or overlap degree is greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles; wherein, the above-mentioned adjacency or overlap degree is the ratio of the first area to the second area, the first area is the sum of the areas of any two circumscribed rectangles in the above-mentioned set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the above-mentioned any two circumscribed rectangles.

[0157] Optionally, those of ordinary skill in the art can understand that Figure 15 the structure shown is only schematic. The electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, and a mobile Internet device (Mobile Internet Devices, MID), a PAD and other terminal devices. Figure 15 It does not limit the structure of the above-mentioned electronic device. For example, the electronic device may further include more or fewer components (such as a network interface, etc.) than those shown in Figure 15 , or have a structure different from that shown in Figure 15The different configurations shown.

[0158] Among them, the memory 1502 can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus in the embodiments of the present invention. The processor 1504 executes various functional applications and data processing by running the software programs and modules stored in the memory 1502, that is, implements the above-mentioned image processing method. The memory 1502 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 1502 may further include a memory remotely disposed relative to the processor 1504, and these remote memories may be connected to the terminal through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and combinations thereof. Among them, the memory 1502 may specifically but not limitedly be used to store information such as the first image and the second image to be compared and the set of target circumscribed rectangles. As an example, as Figure 15 shown, the above-mentioned memory 1502 may but not limitedly include the acquisition unit 1402, the first determination unit 1404, and the merging unit 1406 in the above-mentioned image processing apparatus. In addition, it may also include but not limited to other module units in the above-mentioned image processing apparatus, which will not be elaborated in this example.

[0159] Optionally, the above-mentioned transmission device 1506 is used to receive or send data via a network. Specific examples of the above network may include a wired network and a wireless network. In one instance, the transmission device 1506 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers through a network cable so as to communicate with the Internet or a local area network. In one instance, the transmission device 1506 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0160] In addition, the above-mentioned electronic device further includes: a display 1508 for displaying the first image and the second image to be compared and the set of target circumscribed rectangles; and a connection bus 1510 for connecting each module component in the above-mentioned electronic device.

[0161] In other embodiments, the above terminal device or server may be a node in a distributed system. Among them, the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting the multiple nodes in the form of network communication. Among them, the nodes may form a peer-to-peer (P2P) network, and any form of computing device, such as electronic devices like servers and terminals, can become a node in the blockchain system by joining the peer-to-peer network.

[0162] According to one aspect of the present application, there is provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above image processing method. Among them, the computer program is set to execute the steps in any one of the above method embodiments when running.

[0163] Optionally, in this embodiment, the above computer-readable storage medium may be set to store a computer program for executing the following steps:

[0164] S1. Obtain the difference region in the first image and the second image; among them, the first image and the second image are images to be compared;

[0165] S2. Respectively determine the minimum circumscribed rectangle of the contour corresponding to each difference region to obtain a set of minimum circumscribed rectangles;

[0166] S3. Merge multiple circumscribed rectangles in the above set of minimum circumscribed rectangles whose adjacent or overlapping degree is greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles; among them, the adjacent or overlapping degree is the ratio of the first area to the second area, the first area is the sum of the areas of any two circumscribed rectangles in the above set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the above any two circumscribed rectangles.

[0167] Optionally, in this embodiment, those of ordinary skill in the art can understand that all or part of the steps in the above various methods can be completed by a program instructing the relevant hardware of the terminal device. The program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disc, etc.

[0168] The above serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0169] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage media. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods in various embodiments of the present invention.

[0170] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0171] In the several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0172] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, the functional units in various embodiments of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0174] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An image processing method, characterized in that, comprising: Obtaining the difference region between the first image and the second image, including: grayscale processing the first image and the second image; determining the pixel points at the same positions in the grayscaled first image and second image, and calculating the structural similarity of each of the pixel points at the same positions; determining the difference region based on the structural similarity; wherein, the first image and the second image are images to be compared; Respectively determining the minimum bounding rectangles of the contours corresponding to each difference region to obtain a set of minimum bounding rectangles; Merging multiple bounding rectangles in the set of minimum bounding rectangles whose adjacent or overlapping degree is greater than or equal to a preset threshold to obtain a set of target bounding rectangles; wherein, the adjacent or overlapping degree is the ratio of the first area to the second area, the first area is the sum of the areas of any two bounding rectangles in the set of minimum bounding rectangles, and the second area is the area of the minimum bounding rectangle of the union of the any two bounding rectangles; Wherein, the step of merging multiple bounding rectangles in the set of minimum bounding rectangles whose adjacent or overlapping degree is greater than or equal to a preset threshold to obtain a set of target bounding rectangles includes: Taking any bounding rectangle in the set of minimum bounding rectangles as the current bounding rectangle, and performing the following operations on the current bounding rectangle until all bounding rectangles are traversed: Comparing the adjacent or overlapping degree between the current bounding rectangle and another bounding rectangle outside the current bounding rectangle; When the adjacent or overlapping degree is greater than or equal to the preset threshold, replacing the current bounding rectangle with the minimum bounding rectangle of the current bounding rectangle and the other bounding rectangle, and deleting the other bounding rectangle from the set of minimum bounding rectangles; When the adjacent or overlapping degree between every two bounding rectangles in the set of minimum bounding rectangles is less than the preset threshold, determining to obtain the set of target bounding rectangles.

2. The method according to claim 1, characterized in that, the method further includes: Sorting each bounding rectangle in the set of minimum bounding rectangles in descending order according to the area of each bounding rectangle; Determining the order of the current bounding rectangle according to the result of the descending order.

3. The method according to claim 1, characterized in that, the step of respectively determining the minimum bounding rectangles of the contours corresponding to each difference region to obtain a set of minimum bounding rectangles includes: Performing binarization processing on each of the difference regions; Determining the contours of the binarized difference regions based on a preset contour search algorithm to obtain a set of difference contours; Taking the minimum bounding rectangles of all the parent contours in the set of difference contours as the set of minimum bounding rectangles.

4. The method according to claim 1, characterized in that, after obtaining the set of target bounding rectangles, further including: Identifying the image information included in each bounding rectangle in the set of target bounding rectangles; Outputting the image information.

5. An image processing apparatus, characterized in that, comprising: An obtaining unit, configured to obtain the difference region between the first image and the second image; wherein, the first image and the second image are images to be compared; The above-mentioned acquisition unit specifically includes: a first processing module for performing grayscale processing on the first image and the second image; a second determination module for determining pixel points at the same positions in the first image and the second image after grayscale processing, and calculating the structural similarity of each of the pixel points at the same positions; a third determination module for determining the difference region based on the structural similarity. A first determination unit for respectively determining the minimum circumscribed rectangles of the contours corresponding to each difference region to obtain a set of minimum circumscribed rectangles. A merging unit for merging multiple circumscribed rectangles in the set of minimum circumscribed rectangles whose adjacency or overlapping degree is greater than or equal to a preset threshold to obtain a set of target circumscribed rectangles; wherein, the adjacency or overlapping degree is the ratio of a first area to a second area, the first area is the sum of the areas of any two circumscribed rectangles in the set of minimum circumscribed rectangles, and the second area is the area of the minimum circumscribed rectangle of the union of the any two circumscribed rectangles. The merging unit includes: An execution module for taking any circumscribed rectangle in the set of minimum circumscribed rectangles as the current circumscribed rectangle, and performing the following operations on the current circumscribed rectangle until all circumscribed rectangles are traversed: A first comparison module for comparing the adjacency or overlapping degree between the current circumscribed rectangle and another circumscribed rectangle outside the current circumscribed rectangle. A replacement module for, when the adjacency or overlapping degree is greater than or equal to the preset threshold, replacing the current circumscribed rectangle with the minimum circumscribed rectangle of the current circumscribed rectangle and the other circumscribed rectangle, and deleting the other circumscribed rectangle from the set of minimum circumscribed rectangles. A first determination module for determining the set of target circumscribed rectangles when the adjacency or overlapping degree between every two circumscribed rectangles in the set of minimum circumscribed rectangles is less than the preset threshold.

6. An electronic device, including a memory and a processor, characterized in that a computer program is stored in the memory, and the processor is configured to execute the method described in any one of claims 1 to 4 through the computer program.

7. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 4 when running.

Images