Image recognition system and image recognition program
The system accelerates image detection by reducing data size and employing two-stage matching rate calculations, addressing slow and inaccurate detection in changing environments, achieving significantly faster and more precise results.
Patent Information
- Application Number
- JP2023076282
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-02
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2043-05-02
AI Technical Summary
Conventional image recognition systems struggle with slow and inaccurate image detection, particularly in applications with rapidly changing screens, making real-time processing difficult.
The system employs data size reduction techniques such as color reduction and downscaling, followed by two-stage matching rate calculations to identify matching image components, using first and second determination units to enhance speed and accuracy.
This approach enables high-speed and highly accurate image detection, achieving three to four times faster processing compared to conventional methods while maintaining precision.
Smart Images

Figure 2024160846000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an image recognition system and an image recognition program capable of detecting a predetermined image from images that are the subject of a search. [Background technology]
[0002] Conventionally, various image recognition technologies for extracting other images that are the same as or related to one image have been used for various purposes. As an example of such an image recognition technology, Patent Document 1 describes a method in which a mobile device takes a predetermined image, performs a predetermined process on the taken image, and transmits the image to a remote server as a query image capable of high-speed communication, and the remote server extracts an image corresponding to the query image from image data stored in the server, extracts information content associated with the image, and transmits the extracted information content to a mobile device.
[0003] Another example of image recognition technology is an image recognition technology that detects a specific image element (object image) from among the image elements that make up the image to be searched (search target image). Such image recognition technology is used for various purposes, such as debugging work performed during game production to detect whether there is a bug in a series of play screens and to fix the detected bug.
[0004] Conventionally, image recognition techniques used for such debugging work have detected object images by, for example, calculating a matching rate, which indicates the degree to which a component contained in a search target image matches an object image, using a specified algorithm, and determining that a component with a matching rate exceeding a specified threshold is the same image as the object image.
[0005] When such image recognition technology is used for debugging work during game development, the matching rate between each component in the search target image and the object image is calculated and identity is determined for a series of game screens that change from moment to moment as the game progresses.However, high-speed calculation processing is required to sequentially calculate the matching rate and determine the identity for this constantly changing game screen, and such high-speed calculation processing has traditionally been achieved by an image recognition system that uses, for example, a GPU (Graphics Processing Unit). [Prior art documents] [Patent documents]
[0006] [Patent Document 1] Special Publication No. 2012-524343 Summary of the Invention [Problem to be solved by the invention]
[0007] However, since image recognition by conventional image recognition systems requires a long time to detect object images, it is difficult to perform image recognition processing on the transition screen of an application in real time while using the application in which the screen transitions at high speed. Therefore, there is a demand for an image recognition system that can perform image detection processing at higher speed and with higher accuracy.
[0008] Therefore, the present invention has been devised in consideration of the above-mentioned problems, and an object of the present invention is to provide an image recognition system and an image recognition program capable of performing image detection processing at high speed and with high accuracy. [Means for solving the problem]
[0009] In order to solve the above problem, the image recognition system of the present invention is characterized in that it comprises a data size reduction unit that reduces the data size of a search target image that is the subject of a search and an object image that is the image to be detected, a first calculation unit that calculates a first matching rate, which is the matching rate between a reduced component, which is an image component included in the search target image whose data size has been reduced, and the object image whose data size has been reduced, a first determination unit that determines that the reduced component whose first matching rate exceeds a first threshold is a provisional identical image that provisionally matches the object image, an extraction unit that extracts an initial component corresponding to the provisional identical image from the search target image, a second calculation unit that calculates a second matching rate, which is the matching rate between the initial component and the object image, and a second determination unit that determines that the initial component whose second matching rate exceeds a second threshold is an identical image that matches the object image.
[0010] In the above image recognition system, it is preferable that the data size reduction unit performs at least one of color reduction processing and downscaling processing on the search target image and the object image.
[0011] In addition, in the above image recognition system, it is preferable that the first judgment unit outputs a first coordinate range, which is a coordinate range of the provisionally identical image in the search target image whose data size has been reduced, as a judgment result, and the extraction unit extracts image components located in a second coordinate range, which is a coordinate range corresponding to the first coordinate range in the search target image, as the initial components.
[0012] In addition, in the above image recognition system, it is preferable that the first judgment unit judges that the provisionally identical images in which at least a portion of the first coordinate range overlap are a single provisionally identical image, and outputs a representative coordinate range as a judgment result.
[0013] In addition, in the above image recognition system, it is preferable that the representative coordinate range is the first coordinate range of the provisionally identical image that overlaps at least a portion of the first coordinate range and in which the first matching rate is equal to or greater than a third threshold value.
[0014] In the above image recognition system, it is preferable that the extraction unit determines that the initial components having at least a portion of the second coordinate range overlap each other as a single initial component, and outputs a representative initial component as an extraction result.
[0015] In addition, in the above-mentioned image recognition system, it is preferable that the representative initial component is an initial component that corresponds to the provisionally identical image among the initial components that overlap at least a portion of the second coordinate range and whose first matching rate is equal to or greater than a fourth threshold value.
[0016] In the above image recognition system, it is preferable that the extraction section sets the second coordinate range to be narrower than the first coordinate range.
[0017] The present invention also provides an image recognition program that causes a computer to detect an object image from a search target image, the program including: a data size reduction step for reducing the data size of the search target image and the object image; a first calculation step for calculating a first matching rate, which is the matching rate between reduced components, which are image components included in the search target image with reduced data size, and the object image with reduced data size; a first determination step for determining that the reduced components with the first matching rate exceeding a first threshold are provisional identical images that provisionally match the object image; an extraction step for extracting initial components corresponding to the provisional identical images from the search target image; a second calculation step for calculating a second matching rate, which is the matching rate between the initial components and the object image; and a second determination step for determining that the initial components with the second matching rate exceeding a second threshold are provisional identical images that match the object image. Effect of the Invention
[0018] According to the present invention having the above-mentioned configuration, it becomes possible to perform image detection processing at high speed and with high accuracy. [Brief description of the drawings]
[0019] [Figure 1] FIG. 2 is a conceptual diagram of an image detection process performed by an image recognition system according to an embodiment of the present invention. [Diagram 2] 1 is a configuration block diagram of an image recognition system according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a functional block diagram of a processor used in the image recognition system according to the embodiment of the present invention. [Figure 4] 4 is a flowchart illustrating an example of processing by the image recognition system according to the embodiment of the present invention. [Diagram 5] 1 is a table comparing the image detection speed of an image recognition system according to an embodiment of the present invention with that of a conventional image recognition system. [Figure 6] 1 is a conceptual diagram showing how an object image is detected from a search target image by an image recognition system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] Hereinafter, an image recognition system according to an embodiment of the present invention will be described with reference to the drawings.
[0021] Fig. 1 is a schematic diagram illustrating an example of image detection performed by an image recognition system according to an embodiment of the present invention. As shown in Fig. 1, the image recognition system according to the embodiment of the present invention is a system for detecting an object image O, which is an image to be detected (i.e., an image that is a template for an image to be subjected to image recognition), from a search target image S, which is a search target.
[0022] Specifically, the image recognition system according to the embodiment of the present invention performs a process of detecting image components that match an object image O as the same image from among various image components included in a search target image S. This detection process is performed by sequentially calculating the matching rate between each component in the search target image S and the object image O and determining the identity based on the matching rate, for example, when the search target image S is an image that changes from moment to moment, such as a game screen. The calculation of the matching rate and the determination of identity will be described later in detail.
[0023] Fig. 2 is a configuration block diagram of an image recognition system 1 according to an embodiment of the present invention. As shown in Fig. 2, the image recognition system 1 according to an embodiment of the present invention is a system realized in, for example, a server, and is configured to include a processor 11 and a memory 12 including one or more computer-readable media. The image recognition system 1 can be connected to various external devices (not shown) such as a display and a keyboard, and can receive input from the external devices during image recognition processing, and can transmit the results of the image recognition processing to the external devices and display them thereon. The image recognition system 1 and the external devices may be connected to each other by wire using a cable, or may be connected to each other by wireless communication.
[0024] The processor 11 is capable of performing image recognition processing based on image data input from an external device, and is also capable of transmitting the results of the image recognition processing to the external device and displaying them thereon. The processor 11 operates based on any operating system (OS).
[0025] The memory 12 includes one or more computer-readable media, and is provided with image recognition software 121 executed by the processor 11, and an image data storage unit 122 that stores image data input from an external device, etc.
[0026] Fig. 3 is a functional block diagram of the processor 11 used in the image recognition system 1 according to the embodiment of the present invention. As shown in Fig. 3, the processor 11 executes image recognition software 121 to function as a data size reduction unit 111, a first calculation unit 112, a first determination unit 113, an extraction unit 114, a second calculation unit 115, and a second determination unit 116.
[0027] The data size reduction unit 111 reduces the data size of the search target image S and the object image O. Specifically, the data size reduction unit 111 performs at least one of a color reduction process for reducing the number of colors in the image and a downscaling process for reducing the number of pixels in the image on the search target image S and the object image O.
[0028] Color reduction is performed by averaging the R, G, and B information and converting it into a 256-level grayscale image of brightness only. By performing color reduction, the size of the search target image S and object image O can be reduced, making it possible to quickly perform image detection. In the image recognition system 1, the user can select whether or not color reduction is required and the type of color reduction to be performed by performing a setting operation.
[0029] Downscaling is a process of reducing the size of the search target image S and the object image O by reducing the number of pixels in the search target image S and the object image O to make them coarser images. In the image recognition system 1, the user can arbitrarily select whether downscaling is necessary and to what extent the number of pixels is to be reduced by performing a setting operation. In downscaling, a smaller number of pixels can reduce the image data size and enable faster image detection, but if the number of pixels is reduced below a predetermined number, the image detection accuracy may decrease and appropriate image detection may not be possible. Therefore, it is necessary to perform downscaling within an appropriate range according to the state of the search target image S and the object image O and the required accuracy and speed of image detection.
[0030] The first calculation unit 112 calculates a first matching rate, which is a matching rate between reduced components, which are image components included in the search target image S whose data size has been reduced by the data size reduction unit 111, and an object image O whose data size has also been reduced by the data size reduction unit 111. The calculation of the first matching rate by the first calculation unit 112 can be performed using any image detection algorithm.
[0031] The first determination unit 113 determines that a post-reduction component with a first matching rate exceeding a predetermined threshold (first threshold) is a provisionally identical image that provisionally matches the object image O. Then, the first determination unit 113 outputs a first coordinate range, which is the coordinate range of the provisionally identical image in the search target image S with reduced data size, as a determination result. Note that the coordinate range refers to a predetermined range based on the horizontal and vertical coordinates of the image, such as X=300-340 and Y=100-150 in the horizontal direction, based on the horizontal (X-axis direction) and vertical (Y-axis) coordinates set for the image.
[0032] The extraction unit 114 extracts the initial components corresponding to the provisionally identical image from the search target image S. The extraction of the initial components corresponding to the provisionally identical image is performed based on the coordinate ranges in each image. Specifically, the extraction unit 114 extracts, as the initial components, image components in the second coordinate range, which is the coordinate range corresponding to the first coordinate range output by the first determination unit 113, in the search target image S. Here, the extraction unit 114 sets the second coordinate range to be narrower than the first coordinate range. By making the second coordinate range narrower than the first coordinate range, that is, by trimming, the file size of the initial components can be reduced to perform image detection at a higher speed, and in addition, when visual effects such as light and flames that are often used in games are generated around the initial components, these visual effects can be removed to perform image detection with higher accuracy.
[0033] The second calculation unit 115 calculates a second matching rate, which is a matching rate between the initial component extracted by the extraction unit 114 and the object image O. As with the calculation of the first matching rate, the calculation of this second matching rate can also be performed using any image detection algorithm.
[0034] The second determination unit 116 determines that an initial component having a second matching rate exceeding a predetermined threshold (second threshold) is an identical image that matches the object image O. Then, for the initial component determined to be an identical image that matches the object image O, the second determination unit 116 extracts position information (X coordinate, Y coordinate) in the search target image S and, if the search target image S is a game screen or the like that changes from moment to moment, time information related to the time at which the initial component was detected within the time range to be searched. The position information and time information extracted by the second determination unit 116 are available for use by other applications and are used, for example, in an application that automatically plays a game, to detect an object image O that serves as an indicator for performing a specific action, making it possible to play the game based on the detected object image O.
[0035] Next, a description will be given of the operation of the image recognition system 1 having the above-mentioned configuration, in which an object image O is detected from a search target image S illustrated in FIG.
[0036] 4 is a flowchart showing an example of processing by the image recognition system 1 according to the embodiment of the present invention. First, when image detection processing is started for a search target image S and an object image O specified by a user, the data size reduction unit 111 reduces the data sizes of the search target image S and the object image O (step S1). Here, the data size reduction unit 111 performs color reduction processing and downscaling processing on the search target image S and the object image O to reduce the data sizes of the search target image S and the object image O.
[0037] Next, the first calculation unit 112 calculates a first matching rate, which is a matching rate between reduced components, which are image components included in the search target image S whose data size has been reduced by the data size reduction unit 111, and the object image O whose data size has also been reduced by the data size reduction unit 111, based on a predetermined image detection algorithm (step S2). Because the data sizes of the search target image S and the object image O have been reduced, the first calculation unit 112 can quickly calculate the first matching rate.
[0038] Next, the first determination unit 113 determines that the reduced components with a first matching rate exceeding the first threshold are provisionally identical images that provisionally match the object image O (step S3), and outputs the first coordinate range of the provisionally identical image in the search target image S with reduced data size as the determination result to the extraction unit 114. This determination of provisionally identical images by the first determination unit 113 may be performed for multiple reduced components. In other words, when multiple reduced components with a first matching rate with the object image O exceeding the first threshold are detected in the search target image S, the first determination unit 113 may output these multiple reduced components as the determination result to the extraction unit 114.
[0039] Next, the extraction unit 114 extracts, from the search target image S, image components in a second coordinate range corresponding to the first coordinate range of the provisionally identical image output by the first determination unit 113 as initial components (step S4).
[0040] Next, the second calculation unit 115 calculates a second matching rate, which is a matching rate between the initial component extracted by the extraction unit 114 and the object image O and is more accurate than the first matching rate, based on a predetermined image detection algorithm (step S5).
[0041] Next, the second determination unit 116 determines that the initial components having the second matching rate exceeding the second threshold value are identical images that match the object image O (step S6). By thus determining that the images are identical, a series of image detection processes by the image recognition system 1 is completed.
[0042] Fig. 5 is a table comparing the image detection speed of the image recognition system 1 according to the embodiment of the present invention and a conventional image recognition system. The table shown in Fig. 5 shows the results of image detection performed by the conventional image recognition system and the image recognition system 1 according to the embodiment of the present invention using a search target image S and an object image O of a plurality of image sizes. Here, the conventional image recognition system and the image recognition system 1 according to the embodiment of the present invention use the same hardware, and the conventional image recognition system uses the pattern matching function of OpenCV as the image detection algorithm.
[0043] As shown in FIG. 5, in the image recognition system 1 according to the embodiment of the present invention, the image detection speed is three to four times faster than in the conventional image recognition system.
[0044] FIG. 6 is a conceptual diagram showing how an object image is detected from a search target image by the image recognition system according to the embodiment of the present invention.
[0045] In the example shown in FIG. 6, four identical images SI are detected in the search target image S. Then, the image recognition system 1 generates, for the detected identical images SI, position information as described above, and, if necessary, time information indicating at what point in time the identical images SI appear in the image that changes from moment to moment. For each identical image SI, this position information is generated as a range of X and Y coordinates, for example, "X coordinate range 180 to 210, Y coordinate range 15 to 50." In addition, the time information is generated as the elapsed time from the start to the detection, for example, "2 hours, 4 minutes, and 12 seconds after the start of the search target image S."
[0046] The position information and time information of the identical images SI generated by the image recognition system 1 are used, for example, to detect images used for various operations in an application that automatically plays a game, and automatic play by the application can be realized. For example, in the case of a shooting game shown in FIG. 6, the object image O is an image showing an attack target, and four identical images SI are detected in the search target image S. By monitoring the play situation, it is possible to check whether the game is operating normally, etc., without requiring a game player, and the inspection can be performed quickly and with reduced labor costs.
[0047] In this way, according to the image recognition system 1 of an embodiment of the present invention, the first stage of image detection is performed quickly while reducing the data size of the search target image S and the object image O, and then the range to be detected is narrowed down by identifying identical images using the results of the first image detection, and then a second stage of highly accurate image detection is performed based on the original image, thereby making it possible to perform high-speed and highly accurate image detection processing.
[0048] In the image detection process by the image recognition system 1 described above, as a result of the determination by the first determination unit 113, a plurality of provisionally identical images in which at least a portion of the first coordinate range overlaps may be generated. In such a case, there is a high probability that all of the provisionally identical images show the same image components. If these provisionally identical images are subjected to the processing by the second calculation unit 115 and the second determination unit 116, there is a risk of causing a decrease in the speed of the image detection process. Therefore, if necessary, the provisionally identical images in which at least a portion of the first coordinate range overlaps may be determined to be a single provisionally identical image, and the duplicated determination results may be eliminated and subjected to the processing by the second calculation unit 115 and the second determination unit 116. In this case, the first determination unit 113 determines that the provisionally identical images in which at least a portion of the first coordinate range overlaps are a single provisionally identical image, and performs a process of outputting the representative coordinate range as a determination result, thereby preventing a decrease in the speed of the image detection process.
[0049] Specifically, as the representative coordinate range, for example, the first coordinate range of a provisionally identical image having a first matching rate equal to or higher than a predetermined threshold (third threshold) among provisionally identical images having at least a portion of the first coordinate range overlapping can be used. This enables the second calculation unit 115 and the second determination unit 116 to perform processing with high accuracy. Note that, as the representative coordinate range, the first coordinate range of a provisionally identical image having the highest first matching rate among provisionally identical images having at least a portion of the first coordinate range overlapping can be used, which also enables the second calculation unit 115 and the second determination unit 116 to perform processing with high accuracy.
[0050] Moreover, the elimination of duplicated determination results may be performed by the extraction unit 114, not by the first determination unit 113. Specifically, the extraction unit 114 determines that initial components having at least a part of the second coordinate ranges that overlap are a single initial component, and outputs a representative initial component as the extraction result, thereby preventing a decrease in the speed of the image detection process.
[0051] At this time, as the representative initial component, for example, among the initial components whose second coordinate ranges at least partially overlap, an initial component whose first matching rate corresponds to a tentative identical image having a predetermined threshold value (fourth threshold value) or more can be used. This also enables the second calculation unit 115 and the second determination unit 116 to perform processing with high accuracy. Note that, among the initial components whose second coordinate ranges at least partially overlap, an initial component whose first matching rate corresponds to a tentative identical image having the highest first matching rate may be used as the representative initial component, which also enables the second calculation unit 115 and the second determination unit 116 to perform processing with high accuracy.
[0052] Furthermore, for such provisionally identical images in which at least a portion of the first coordinate ranges overlap, all of the overlapping provisionally identical images may be subjected to processing by the second calculation unit 115 and the second determination unit 116 without eliminating the overlapping determination results. By performing such processing, the accuracy of image detection can be improved.
[0053] The present invention is not limited to the above-described embodiment, and various modifications can be made. [Explanation of symbols]
[0054] 1: Image recognition system 11: Processor 12: Memory 121: Image recognition software 122: Image data storage unit 111: Data size reduction section 112: First calculation unit 113:First judgment part 114:Extraction part 115: Second calculation unit 116:Second judgment part O: Object image S: Search target image SI: Same image
Claims
1. a data size reduction unit that reduces the data size of a search target image that is a search target and an object image that is an image to be detected; a first calculation unit that calculates a first matching rate that is a matching rate between reduced components that are image components included in the search target image whose data size has been reduced and the object image whose data size has been reduced; a first determination unit that determines that the post-reduction component having the first matching rate exceeding a first threshold is a provisionally identical image that provisionally matches the object image; an extraction unit that extracts an initial component corresponding to the provisional identical image from the search target image; a second calculation unit that calculates a second matching rate, which is a matching rate between the initial component and the object image; a second determination unit that determines that the initial component having the second matching rate exceeding a second threshold is an identical image that matches the object image; An image recognition system comprising:
2. The image recognition system according to claim 1 , wherein the data size reduction unit performs at least one of a color reduction process and a downscaling process on the search target image and the object image.
3. the first determination unit outputs, as a determination result, a first coordinate range that is a coordinate range of the provisionally identical image in the search target image whose data size has been reduced; The image recognition system according to claim 1 , wherein the extraction unit extracts, as the initial components, image components that are in a second coordinate range that is a coordinate range corresponding to the first coordinate range in the image to be searched.
4. The image recognition system according to claim 3 , wherein the first determination unit determines that the provisionally identical images in which at least a portion of the first coordinate ranges overlap are a single provisionally identical image, and outputs a representative coordinate range as a determination result.
5. The image recognition system according to claim 4 , wherein the representative coordinate range is the first coordinate range of the provisionally identical image that overlaps at least a portion of the first coordinate range and in which the first matching rate is equal to or greater than a third threshold value.
6. The image recognition system according to claim 3 , wherein the extraction unit determines that the initial components whose second coordinate ranges at least partially overlap are a single initial component, and outputs a representative initial component as an extraction result.
7. The image recognition system according to claim 6 , wherein the representative initial component is an initial component that corresponds to the provisionally identical image and has a first matching rate equal to or greater than a fourth threshold value, among the initial components that overlap at least a portion of the second coordinate range.
8. The image recognition system according to claim 3 , wherein the extraction unit sets the second coordinate range to be narrower than the first coordinate range.
9. An image recognition program that causes a computer to detect an object image from a search target image, a data size reduction step of reducing a data size of the search target image and the object image; a first calculation step of calculating a first matching rate which is a matching rate between reduced components which are image components included in the search target image whose data size has been reduced and the object image whose data size has been reduced; a first determination step of determining that the post-reduction component having the first matching rate exceeding a first threshold is a provisionally identical image that provisionally matches the object image; an extraction step of extracting an initial component corresponding to the provisional identical image from the search target image; a second calculation step of calculating a second matching rate which is a matching rate between the initial component and the object image; a second determination step of determining that the initial component having the second matching rate exceeding a second threshold is an identical image that matches the object image; 2. An image recognition program comprising:
Citation Information
Patent Citations
Pattern matching device
JP1988004379A
Image searching device
JP1993159064A
pattern matching method
JP1993288520A
Device and method for discriminating image
JP2003331288A
Image processor detecting / recognizing moving body
JP2005173787A
