Image recognition method, device and storage medium

By intercepting preset area images during video playback and matching feature points to achieve video recognition, the problem of deep convolutional neural network occupies a large resource, real-time and efficient image recognition is achieved.

CN110826355BActive Publication Date: 2024-12-03TENCENT DIGITAL TIANJIN
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN201810890946.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2018-08-07
Publication Date
2024-12-03
Estimated Expiration
2038-08-07

AI Technical Summary

Technical Problem

The prior art uses deep convolutional neural networks to perform image recognition during video playback, and occupies a large amount of computing resources, especially when the video playback volume is large, which affects the system performance and makes it difficult to realize real-time image recognition.

Method used

By intercepting the preset area image in the video screen, obtain the feature point set and feature point descriptors, match the feature points of the reference image according to the Hanming distance, detect the matching of the feature point. If the match is successful, determine the image matching and reduce the amount of data calculation And improve the recognition speed.

Benefits of technology

Real-time image recognition on video platforms or user terminals is realized, resource occupation is reduced, operation and maintenance costs are saved, image recognition is improved, and it is suitable for scenes with large-scale video playback volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110826355B_ABST
    Figure CN110826355B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention discloses an image recognition method, device and storage medium. The embodiment of the present invention can intercept an image of a preset area in a video screen during video playback; obtain a feature point set of the image and a descriptor of the feature point in the feature point set; obtain a reference point with the shortest Hamming distance to the feature point in a preset reference image feature set according to the descriptor of the feature point; detect whether the feature point matches the reference point correctly; if the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully. Compared with the existing technology, this solution occupies less resources and reduces the burden of data processing while ensuring the efficiency of image recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular to an image recognition method, device and storage medium. Background Art

[0002] With the development of network technology, live streaming has developed rapidly as a new form of online social interaction. Users can live stream e-sports games, movies, TV series, etc., and can also watch other netizens' live streams.

[0003] In live broadcasts, image recognition technology can be used to identify exciting events or plots to remind viewers to watch and enhance the viewing experience. For example, in live broadcasts of e-sports games, when exciting game events such as "pentakill" or "quadruplekill" occur, this information can be obtained through image recognition to promptly remind viewers not to miss the exciting moments. Currently, the mainstream solution for image recognition in videos is to use a deep convolutional neural network training model for image recognition.

[0004] In the process of research and practice of the prior art, the inventors of the present invention found that during the video playback process, using a deep convolutional neural network training model for image recognition requires a large amount of computing resources. Especially when the video playback volume of the video playback platform increases, performing image recognition on a large number of videos online at the same time will aggravate resource usage and affect system performance. Summary of the invention

[0005] The embodiments of the present invention provide an image recognition method, device and storage medium, aiming to solve the technical problem that image recognition in a video occupies a large amount of resources.

[0006] This method embodiment provides an image recognition method, including:

[0007] During video playback, capturing an image of a preset area in the video screen;

[0008] Acquire a feature point set of the image and descriptors of feature points in the feature point set;

[0009] According to the descriptor of the feature point, obtaining a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set;

[0010] Detecting whether the feature point matches the reference point correctly;

[0011] If the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully.

[0012] In some embodiments, obtaining a feature point set of the image includes:

[0013] Performing feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image;

[0014] The feature points in the candidate feature point set are screened according to preset rules to obtain the feature point set of the image.

[0015] In some embodiments, the step of screening the feature points in the candidate feature point set according to a preset rule to obtain the feature point set of the image includes:

[0016] Calculate the corner point response value of each feature point in the candidate feature point set respectively;

[0017] A preset number of feature points are screened out from the candidate feature point set according to the corner point response values, and a feature point set of the image is obtained by configuration.

[0018] In some embodiments, the step of selecting a preset number of feature points from the candidate feature point set according to the corner point response value to obtain the feature point set of the image further includes:

[0019] The required quantity of feature points is determined according to the position of the preset area, and the required quantity is configured as the preset quantity.

[0020] In some embodiments, obtaining a feature point set of the image includes:

[0021] Taking a pixel point in the image as the center of a circle, determining a reference point corresponding to the pixel point according to a preset radius;

[0022] Obtaining grayscale value information of the pixel and its corresponding reference point, and determining whether the pixel point is a feature point according to the grayscale value information;

[0023] If the pixel point is a feature point, the pixel point is configured into a feature point set of the image.

[0024] In some embodiments, the step of obtaining a feature point set of the image and a descriptor of a feature point in the feature point set further comprises:

[0025] Obtaining a hash string of the image, and calculating a Hamming distance between the hash string and a preset reference image hash string;

[0026] If the Hamming distance is less than a preset first threshold, executing the step of acquiring a feature point set of the image and a descriptor of the feature point in the feature point set;

[0027] If the Hamming distance exceeds a preset first threshold, it is determined that the image fails to match the reference image.

[0028] In some embodiments, the detecting whether the feature point matches the reference point correctly includes:

[0029] Acquire a neighborhood point set of the feature point as a first neighborhood point set, and acquire a neighborhood point set of the reference point as a second neighborhood point set;

[0030] Matching the pixel points in the first neighborhood point set and the second neighborhood point set, and obtaining the number of successfully matched pixel point pairs;

[0031] If the number of successfully matched pixel pairs exceeds a preset second threshold, it is determined that the feature point and the reference point are correctly matched.

[0032] In some embodiments, during the video playback, capturing an image of a preset area in the video screen includes:

[0033] Acquire a preset reference image and its corresponding preset area;

[0034] Feature point recognition is performed on the reference image to obtain a reference image feature set.

[0035] The embodiment of the present invention further provides an image recognition device, comprising:

[0036] A screenshot unit, used to capture an image of a preset area in the video screen during video playback;

[0037] A feature unit, used to obtain a feature point set of the image and a descriptor of a feature point in the feature point set;

[0038] A matching unit, configured to obtain, according to the descriptor of the feature point, a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set;

[0039] A detection unit, used to detect whether the feature point matches the reference point correctly;

[0040] The determination unit is used to determine that the image matches the reference image successfully if the number of correctly matched feature points in the feature point set exceeds a preset value.

[0041] In some embodiments, the feature unit includes a candidate subunit and a screening subunit:

[0042] The candidate subunit is used to perform feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image;

[0043] The screening subunit is used to screen the feature points in the candidate feature point set according to a preset rule to obtain the feature point set of the image.

[0044] In some embodiments, the screening subunit is specifically used for:

[0045] The corner point response value of each feature point in the candidate feature point set is calculated respectively; a preset number of feature points are screened out from the candidate feature point set according to the corner point response value, and a feature point set of the image is obtained by configuration.

[0046] In some embodiments, the feature unit further includes a quantity subunit:

[0047] The quantity subunit is used to determine the required quantity of feature points according to the position of the preset area, and configure the required quantity to the preset quantity.

[0048] In some embodiments, the feature unit includes an identification subunit:

[0049] The identification subunit is used to determine the reference point corresponding to the pixel point in the image according to a preset radius with the pixel point in the image as the center of the circle; obtain the gray value information of the pixel and its corresponding reference point, and determine whether the pixel point is a feature point according to the gray value information; if the pixel point is a feature point, configure the pixel point into the feature point set of the image.

[0050] In some embodiments, the apparatus further comprises a hashing unit:

[0051] The hash unit is used to obtain a hash string of the image, and calculate a Hamming distance between the hash string and a preset reference image hash string; if the Hamming distance exceeds a preset first threshold, it is determined that the image fails to match the reference image;

[0052] The feature unit is further configured to obtain a feature point set of the image and a descriptor of a feature point in the feature point set if the Hamming distance is less than a preset first threshold.

[0053] In some embodiments, the detection unit is specifically used to:

[0054] The neighborhood point set of the feature point is obtained as a first neighborhood point set, and the neighborhood point set of the reference point is obtained as a second neighborhood point set; pixel points in the first neighborhood point set and the second neighborhood point set are matched, and the number of successfully matched pixel point pairs is obtained; if the number of successfully matched pixel point pairs exceeds a preset second threshold, it is determined that the feature point is correctly matched with the reference point.

[0055] In some embodiments, the apparatus further comprises a reference unit:

[0056] The reference unit is used to obtain a preset reference image and a corresponding preset area; perform feature point recognition on the reference image to obtain a reference image feature set.

[0057] An embodiment of the present invention further provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the steps in any image recognition method provided by the embodiment of the present invention.

[0058] As can be seen from the above, in the process of video playback, the embodiment of the present invention intercepts the image of the preset area in the video screen to perform image recognition; then, obtains the feature point set of the image and the descriptor of the feature point in the feature point set; then, according to the descriptor of the feature point, obtains the reference point with the shortest Hamming distance to the feature point in the preset reference image feature set, and detects whether the feature point and the reference point are correctly matched to eliminate false matches and ensure the accuracy of image recognition; if the number of correctly matched feature points in the feature point set exceeds the preset value, it can be considered that the image matches the reference image, and the recognition of the reference image is achieved. This solution does not need to use a complex training model. By matching the feature points of the image in a specific area with the reference points in the reference image feature set, it is determined whether the image in a specific area is a reference image, which reduces the amount of data calculation, speeds up the image recognition processing speed, and realizes the rapid recognition of the specified image. In addition, in this solution, after obtaining the reference point with the shortest Hamming distance between the feature point and it, the correct matching detection of the feature point and the reference point with the shortest Hamming distance between it can effectively eliminate false matches and improve the accuracy of image recognition. In addition, this solution can perform image recognition on videos in real time in the background of the video platform or in the user terminal. The increase in the amount of video playback on the video platform will not affect the accuracy and speed of image recognition. The application is more flexible and convenient, saving a lot of cloud resources and operation and maintenance costs. Therefore, compared with existing technologies, this solution occupies less resources, reduces the burden of data processing, and ensures the efficiency of image recognition, and can realize real-time image recognition of video images. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1a is a schematic diagram of a scenario of an information interaction system provided by an embodiment of the present invention;

[0061] Figure 1b is a flowchart of an image recognition method provided by an embodiment of the present invention;

[0062] Figure 1c is a schematic diagram of an application scenario of feature point recognition provided by an embodiment of the present invention;

[0063] Figure 2 is a flowchart of another image recognition method provided by an embodiment of the present invention;

[0064] Figure 3a is a flowchart of another image recognition method provided by an embodiment of the present invention;

[0065] Figure 3b is a schematic diagram of a picture including a reference image provided by an embodiment of the present invention;

[0066] Figure 3c is a schematic diagram of characteristic points provided by an embodiment of the present invention;

[0067] Figure 3d is a schematic diagram of a video screen provided by an embodiment of the present invention;

[0068] Figure 3e is a schematic diagram of correct matching of feature points and reference points provided by an embodiment of the present invention;

[0069] Figure 4 is a schematic diagram of the structure of an image recognition device provided by an embodiment of the present invention;

[0070] Figure 5 is another structural schematic diagram of an image recognition device provided by an embodiment of the present invention;

[0071] Figure 6 Schematic diagram of another structure of the image recognition device provided by the embodiment of the present invention

[0072] Figure 7 is a schematic diagram of the structure of a server provided by an embodiment of the present invention;

[0073] Figure 8 It is a schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0075] The embodiment of the present invention provides an information interaction system, which includes an image device provided by any embodiment of the present invention, and the image recognition device can be integrated in a client, a server, or a third-party network device, etc. In addition, the system can also include other devices, such as a client and a server, etc. The client can be a terminal or a personal computer (PC, Personal Computer) and other devices. The server can be a server and other devices.

[0076] refer to Figure 1a , an embodiment of the present invention provides an information interaction system, including an image recognition device (not shown in the figure), a client and a server. The server is used to provide video data, such as live video; the client is used to play the video; the image recognition device is used to perform image recognition on the video. This embodiment takes the image recognition device integrated in the client as an example. The client and the server are connected through a network or through a core router. The core router forwards the video data sent by the server to the client, and the image recognition device in the client performs image recognition on the video screen.

[0077] During the video playback process on the client, the image recognition device captures the image of a preset area in the video screen for image recognition; then, obtains the feature point set of the image and the descriptors of the feature points in the feature point set; then, based on the descriptors of the feature points, obtains the reference point with the shortest Hamming distance to the feature point in the preset reference image feature set, and detects whether the feature point and the reference point are correctly matched to eliminate false matches and ensure the accuracy of image recognition; if the number of correctly matched feature points in the feature point set exceeds the preset value, it can be considered that the image matches the reference image, and the recognition of the reference image is achieved.

[0078] This solution does not require the use of complex training models. By matching the feature points of the image in a specific area with the reference points in the feature set of the reference image, it determines whether the image in a specific area is a reference image, which reduces the amount of data calculation, speeds up the image recognition processing speed, and realizes the rapid recognition of the specified image. In addition, after obtaining the reference point with the shortest Hamming distance between the feature point and its reference point, the solution also performs correct matching detection on the reference point with the shortest Hamming distance between the feature point and its reference point, which can effectively eliminate false matches and improve the accuracy of image recognition. In addition, this solution can perform image recognition on the video in real time in the background of the video platform or in the user terminal. The increase in the amount of video playback in the video platform will not affect the accuracy and speed of image recognition. The application is more flexible and convenient, saving a lot of cloud resources and saving operation and maintenance costs. Therefore, compared with the existing technology, this solution has a small resource footprint, reduces the burden of data processing, and ensures the efficiency of image recognition, and can realize real-time image recognition of video images.

[0079] Above Figure 1aThe example is only an example of a system architecture for implementing the embodiment of the present invention. The embodiment of the present invention is not limited to the above Figure 1a The system structure shown is based on which various embodiments of the present invention are proposed.

[0080] In this embodiment, the description will be made from the perspective of an image recognition device, which may be integrated into a network device such as a terminal or a server.

[0081] An embodiment of the present invention provides a distributed denial of service attack protection method, comprising: during video playback, capturing an image of a preset area in a video screen; obtaining a feature point set of the image, and descriptors of the feature points in the feature point set; according to the descriptors of the feature points, obtaining a reference point with the shortest Hamming distance to the feature point in a preset reference image feature set; detecting whether the feature point matches the reference point correctly; and determining that the image matches the reference image successfully if the number of correctly matched feature points in the feature point set exceeds a preset value.

[0082] like Figure 1b As shown, the specific process of the image recognition method can be as follows:

[0083] 101. During video playback, capture an image of a preset area in the video screen.

[0084] For example, when broadcasting live, or playing movies, TV and other videos, the image recognition device integrated in the playback background can perform image recognition on the video. Of course, when watching live broadcasts, or playing movies, TV and other videos, the image recognition device integrated in the client can also perform image recognition on the video. When a specified event that needs to be identified occurs in the video screen, such as exciting game events such as "penta kill" or "quadruple kill", the corresponding image will be displayed, such as the words "penta kill" or "quadruple kill", etc. Therefore, these images with the words "penta kill" or "quadruple kill" can be used as reference images to perform image recognition in the video screen. If the reference image is recognized to appear in the video screen, it can be determined that the specified event has occurred.

[0085] During video playback, the image recognition device can capture an image of a preset area in the video screen according to a preset frequency to perform image recognition. The preset area can be configured according to the position or area displayed by the reference image to be detected. In addition, the preset area can be configured based on experience of the position displayed by the reference image, or can be configured based on a fixed position displayed by the reference image, and can be flexibly configured or adjusted.

[0086] After obtaining the image of the preset area, the image recognition device can pre-process the image, including: scaling the image to a preset size and / or grayscale processing the image, etc. The execution order of image scaling and grayscale processing is not limited and can be flexibly configured according to actual needs. Scaling the image to a preset size can standardize the image to avoid the impact of differences caused by different image sizes on the recognition results; grayscale processing the image, then the pixels in the image can be represented by grayscale values. The description of the grayscale image still reflects the distribution and characteristics of the overall and local chromaticity and brightness levels of the entire image, just like the color image, but it can reduce the amount of calculation for subsequent image recognition compared to the color image.

[0087] For example, the preset frequency may be 1 time per second, and the image recognition device will capture an image of a preset area in the video screen every second for detection.

[0088] If multiple reference images need to be recognized during video playback, the image recognition device will capture images of preset areas corresponding to the reference images respectively to perform image recognition.

[0089] 102. Obtain a feature point set of the image and descriptors of the feature points in the feature point set.

[0090] After obtaining the image of the preset area, the image recognition device recognizes feature points of the image and configures the recognized feature points into a set to obtain a feature point set of the image. Then, each feature point in the feature point set is described to obtain a descriptor of each feature point.

[0091] As an implementation method, the step of "obtaining a feature point set of an image" may include: taking a pixel point in the image as the center of a circle, determining a reference point corresponding to the pixel point according to a preset radius; obtaining grayscale value information of the pixel and its corresponding reference point, and determining whether the pixel point is a feature point according to the grayscale value information; if the pixel point is a feature point, configuring the pixel point into the feature point set of the image.

[0092] The image recognition device can detect whether each pixel in the image is a feature point. Specifically, the image recognition device takes a pixel in the image, takes the pixel as the center, draws a circle on the image according to a preset radius, and takes the pixel passing through the circle as the reference point corresponding to the pixel.

[0093] Then, the image recognition device obtains the gray value information of this pixel and its corresponding reference point. If the gray values ​​of n consecutive reference points are greater than the reference point, or the gray values ​​of n consecutive reference points are less than the reference point, then this pixel can be determined as a feature point. Where n is a preset value, which can be flexibly configured according to actual needs.

[0094] Of course, in order to ensure the accuracy of the feature points, a threshold can also be set to filter out reference points that are not much different from the pixel points. For example, the grayscale value of this pixel point is I, and the threshold is t. If there are more than n consecutive pixel points on the circumference whose grayscale values ​​are greater than I+t, or smaller than It, then this pixel point is determined to be a feature point, otherwise it is not.

[0095] Then, the feature point is configured into the feature set of the image. Similarly, the image recognition device detects whether each pixel in the image is a feature point, and configures all the feature points into the feature set of the image, thereby obtaining the feature set of the image.

[0096] For example, refer to Figure 1c , the image recognition device detects whether pixel point 0 in the image is a feature point, taking the preset radius as 3 pixels and n as 12.

[0097] Then, the image recognition device draws a circle with pixel 0 as the center and 3 pixels as the radius. The circumference of the circle passes through 16 pixels in total. The image recognition device uses these 16 pixels as reference points corresponding to pixel 0, and numbers these 16 reference points in chronological order. First, the image recognition device can detect the grayscale values ​​of reference points 1, 5, 9, and 13. If three or more of the grayscale values ​​of these four reference points are greater than or less than the grayscale value of pixel 0, the image recognition device further detects whether there are 12 consecutive pixel values ​​greater than the pixel value of pixel 0, or whether there are 12 consecutive pixel values ​​less than the pixel value of pixel 0. If so, the image recognition device determines that pixel 0 is a feature point and configures it into the feature point set of the image.

[0098] If there are not three or more grayscale values ​​of reference points 1, 5, 9 and 13 that are greater than or less than the grayscale value of pixel 0, then pixel 0 is determined not to be a feature point. This achieves a preliminary judgment of the feature point, reduces the amount of data calculation, and improves the judgment speed.

[0099] After obtaining the feature set of the image, the image recognition device needs to obtain the descriptor of each feature point in the feature set, wherein the descriptor is a binary string or matrix that describes the characteristics of the feature point.

[0100] As an implementation method, the image recognition device may use the BRIEF (Binary Robust Independent Elementary Features) algorithm to obtain the descriptors of each feature. Specifically, the image recognition device selects m pairs of pixel points pi and qi (i=1, 2, ..., m) in the neighborhood of a feature point according to a preset mode, the grayscale value of Pi is I(pi), and the grayscale value of qi is I(qi). Then, the image recognition device compares the grayscale values ​​of each pair of pixels. If I(pi)>I(qi), the image recognition device generates a return value of 1, otherwise the return value is 0. When all pixel pairs are compared, the image recognition device configures the obtained return value as a binary string of length m according to a preset order as the descriptor of this feature point. By analogy, the image recognition device obtains the descriptors of each feature point in the feature point set respectively.

[0101] 103. According to the descriptor of the feature point, a reference point having the shortest Hamming distance to the feature point is obtained from a preset reference image feature set.

[0102] The preset reference image feature set includes reference points for feature point matching, wherein each reference point is characterized by a descriptor. It should be noted that the preset reference image feature set is configured in advance according to the reference image to be identified, wherein each reference point is a feature point in the reference image. Moreover, the length of the descriptor string of each reference point is consistent with the length of the feature point descriptor string obtained in step 102. The reference image is the target image to be identified, which can be user-defined or preset by the video playback platform.

[0103] The Hamming distance between two strings of equal length refers to the number of different characters in the corresponding positions of the two strings. In other words, it is the number of characters that need to be replaced to transform one string into another. The image recognition device uses brute force enumeration based on the descriptors of each feature point of the image to obtain the Hamming distance with each reference point in the reference image feature set, and then selects the reference point with the shortest Hamming distance. Among them, the reference point with the shortest Hamming distance to the feature point is also the reference point that is most similar to the feature point.

[0104] Thus, the image recognition device obtains the reference point with the shortest Hamming distance between each feature point.

[0105] 104. Check whether the feature points and reference points are correctly matched.

[0106] In order to eliminate false matches, the image recognition device may further detect whether the feature point correctly matches the reference point with the shortest Hamming distance therebetween.

[0107] Specifically, as an implementation method, step 104 may include: obtaining a neighborhood point set of the feature point as a first neighborhood point set, and obtaining a neighborhood point set of the reference point as a second neighborhood point set; matching pixel points in the first neighborhood point set and the second neighborhood point set, and obtaining the number of successfully matched pixel point pairs; if the number of successfully matched pixel point pairs exceeds a preset second threshold, it is determined that the feature point and the reference point are correctly matched.

[0108] If the feature point is correctly matched with the reference point with the shortest Hamming distance, the pixels around the two should also be the same. Therefore, the matching of the neighborhood pixels can be used to detect whether the two are correctly matched. For example, GMS (Grid-based Motion Statistics) can be used to detect whether the feature point is correctly matched with the reference point with the shortest Hamming distance.

[0109] The image recognition device can extract the neighborhood pixel points of the feature point according to a preset size, such as 3×3, configure the neighborhood point set of the feature point, and use the neighborhood point set of the feature point as the first neighborhood point set. In addition, the image recognition device obtains the neighborhood pixel points of the reference point according to the preset size, configures the neighborhood point set of the reference point, and uses the neighborhood point set of the reference point as the second neighborhood point set. It should be noted that when extracting neighborhood pixel points, the positional relationship between the feature point and its neighborhood pixel points is the same as the positional relationship between the reference point and its neighborhood pixel points. Therefore, the pixels in the first neighborhood point set and the second neighborhood point set are one-to-one corresponding in position. For example, the feature point can be used as the center point, and the neighborhood pixel points around it can be extracted according to the preset size to obtain the first neighborhood point set. Similarly, the reference point can be used as the center point, and the neighborhood pixel points around it can be extracted according to the preset size to obtain the second neighborhood point set.

[0110] The image recognition device may extract the neighborhood pixel points of the feature point in the captured image, and extract the neighborhood pixel points of the reference point in the reference image. Alternatively, the image recognition device may extract the neighborhood pixel points of the feature point in the feature point set, and extract the neighborhood pixel points of the reference point in the reference image feature set, so as to improve the accuracy of matching. The specific configuration may be flexible according to actual needs.

[0111] After obtaining the first neighborhood point set and the second neighborhood point set, the image recognition device detects whether the pixels at the same position in the first neighborhood point set and the second neighborhood point set are the same. Specifically, the pixels at the same position in the first neighborhood point set and the second neighborhood point set are taken as pixel pairs, descriptors of the two pixels in the pixel pair are obtained, and the Hamming distance of the two pixels in the pixel pair is obtained according to the descriptors. If the Hamming distance is less than a preset value, it is determined that the pixel pair is matched successfully.

[0112] The image recognition device counts the number of successfully matched pixel pairs. If the number of successfully matched pixel pairs exceeds a preset second threshold, it means that the feature point and the surrounding pixels of the reference point with the shortest Hamming distance are substantially the same, and the image recognition device can determine that the feature point is correctly matched with the reference point with the shortest Hamming distance.

[0113] Thus, the image recognition device determines whether each feature point in the feature point set correctly matches the reference point with the shortest Hamming distance thereto, and obtains a detection result.

[0114] 105. If the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully.

[0115] The image recognition device counts the number of feature points in the feature point set that are correctly matched with the reference point with the shortest Hamming distance.

[0116] Since the reference image corresponds to the reference image feature set, each reference point in the reference image feature set is extracted from the reference image. Therefore, if the number of correctly matched feature points exceeds the preset value, it can be determined that the image in the preset area of ​​the video screen is highly similar to the reference image, and the match is successful, thereby realizing the recognition of the reference image.

[0117] In some embodiments, after determining that the image of the preset area in the video screen successfully matches the reference image, the image recognition device can determine that a reference event corresponding to the reference image has occurred in the current video, and then display a prompt message corresponding to the reference event in the video screen, and can also be assisted by a preset prompt sound. Then, the image recognition device continues to perform step 101, intercepting the image of the preset area in the video screen for image recognition.

[0118] In some embodiments, if the number of correctly matched feature points in the feature point set does not exceed a preset value, it can be determined that the image of the preset area in the video screen fails to match the reference image, and the image recognition device continues to execute step 101 to capture the image of the preset area in the video screen for image recognition.

[0119] As can be seen from the above, in the process of video playback, the embodiment of the present invention intercepts the image of the preset area in the video screen to perform image recognition; then, obtains the feature point set of the image and the descriptor of the feature point in the feature point set; then, according to the descriptor of the feature point, obtains the reference point with the shortest Hamming distance to the feature point in the preset reference image feature set, and detects whether the feature point and the reference point are correctly matched to eliminate false matches and ensure the accuracy of image recognition; if the number of correctly matched feature points in the feature point set exceeds the preset value, it can be considered that the image matches the reference image, and the recognition of the reference image is achieved. This solution does not need to use a complex training model. By matching the feature points of the image in a specific area with the reference points in the reference image feature set, it is determined whether the image in a specific area is a reference image, which reduces the amount of data calculation, speeds up the image recognition processing speed, and realizes the rapid recognition of the specified image. In addition, in this solution, after obtaining the reference point with the shortest Hamming distance between the feature point and it, the correct matching detection of the feature point and the reference point with the shortest Hamming distance between it can effectively eliminate false matches and improve the accuracy of image recognition. In addition, this solution can perform image recognition on videos in real time in the background of the video platform or in the user terminal. The increase in the amount of video playback on the video platform will not affect the accuracy and speed of image recognition. The application is more flexible and convenient, saving a lot of cloud resources and operation and maintenance costs. Therefore, compared with existing technologies, this solution occupies less resources, reduces the burden of data processing, and ensures the efficiency of image recognition, and can realize real-time image recognition of video images.

[0120] According to the method described in the previous embodiment, the image recognition device is specifically integrated in a terminal as an example for further detailed description.

[0121] In one embodiment, an image recognition method is provided, which is different from the above-mentioned image recognition method embodiment in that: in this embodiment, the terminal screens the feature points in the feature point set, which can further reduce the amount of data calculation and improve the accuracy of image recognition. Figure 2 , the method comprising:

[0122] 201. During video playback, the terminal captures an image of a preset area in the video screen.

[0123] For example, when a user uses a terminal to watch a live video, the terminal can capture an image of a preset area in the video screen at preset time intervals during the video playback process for image recognition.

[0124] 202. The terminal performs feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image.

[0125] The terminal identifies the feature points in the image and configures a set of candidate feature points corresponding to the image.

[0126] Specifically, as an implementation method, step 202 may include: the terminal uses the pixel point in the image as the center of the circle, and determines the reference point corresponding to the pixel point according to a preset radius; obtains the grayscale value information of the pixel and its corresponding reference point, and determines whether the pixel point is a feature point according to the grayscale value information; if the pixel point is a feature point, the pixel point is configured into the candidate feature point set of the image.

[0127] The specific implementation method can refer to the description of step 102 in the above-mentioned image recognition method embodiment, which will not be repeated here.

[0128] 203. The terminal screens feature points in the candidate feature point set according to a preset rule to obtain a feature point set of the image.

[0129] After obtaining the candidate feature point set, the terminal selects some feature points from the candidate feature point set according to preset rules, and configures the selected feature points into the feature point set of the image.

[0130] Among them, the preset rules can be to randomly screen a preset number of feature points, or to screen out feature points at preset positions in the image. Of course, it can also be a rule to screen out feature points whose corner point response values ​​exceed preset values, so that the screened feature points are more representative, which can not only reduce the calculation amount of feature point matching, but also improve the accuracy of image recognition.

[0131] In some embodiments, the preset rule is to filter out a preset number of corner points, and step 203 may include: the terminal calculates the corner point response value of each feature point in the candidate feature point set respectively; based on the corner point response value, filters out a preset number of feature points in the candidate feature point set, and configures a feature point set of the image.

[0132] The preset number can be flexibly configured according to actual needs, such as 200 or 600. In this embodiment, the preset number is x.

[0133] In some embodiments, the terminal may use the Harris algorithm to calculate the corner point response value of each feature point in the candidate feature point set.

[0134] Then, the terminal sorts the feature points in the candidate feature point set according to the corner point response value to obtain a feature point sequence whose corner point response values ​​are arranged from large to small. The terminal selects the feature points ranked from the 1st to the xth in the feature point sequence as the screened corner points. The terminal configures these x feature points into the feature point set, and obtains the feature point set of the image.

[0135] Therefore, the response values ​​of the corner points of the feature points in the screened feature point set are relatively large and the features are obvious, which can improve the accuracy of feature point matching.

[0136] In some embodiments, the number of feature points in the feature point set can be configured according to the properties of the image to improve the flexibility and accuracy of feature point screening. Specifically, before the step of "screening a preset number of feature points from the candidate feature point set according to the corner point response value, and configuring the feature point set of the image", it can also include: the terminal determines the required number of feature points according to the position of the preset area, and configures the required number to the preset number.

[0137] When the terminal performs image recognition, preset areas at different positions correspond to different images that need to be recognized. For example, images such as text and hero portraits in games have different corresponding preset area positions.

[0138] Therefore, the terminal can determine the corresponding reference image to be recognized according to the preset area position of the captured image, and further determine the required number of feature points when performing image recognition according to the category of the reference image.

[0139] For example, if the reference image corresponding to the preset area is text and its category is text, and its image edge is relatively obvious, the demand for feature points can be relatively small during image recognition, which not only reduces the amount of feature point matching calculations, but also ensures the accuracy of image recognition.

[0140] If the reference image corresponding to the preset area is a game hero portrait, and the category is hero, the edge of the image may not be so obvious. In this case, when performing image recognition, the demand for feature points can be relatively large, which slightly reduces the amount of feature point matching calculations and ensures the accuracy of image recognition.

[0141] As a result, it is possible to filter a specific number of feature points for matching according to the category of the image, thereby improving the flexibility of image recognition.

[0142] 204. The terminal obtains descriptors of feature points in the feature point set.

[0143] The terminal may use the BRIEF algorithm to obtain the descriptor of each feature. The specific implementation method may refer to the description of step 102 in the above-mentioned image recognition method embodiment, which will not be repeated here.

[0144] 205. The terminal obtains a reference point having the shortest Hamming distance to the feature point from a preset reference image feature set according to the descriptor of the feature point.

[0145] The terminal uses brute force enumeration to find the reference point with the shortest Hamming distance to each characteristic diagnosis point in the feature point set in the preset reference image set, thereby obtaining the reference point that is most similar to each characteristic point.

[0146] The preset reference image feature set is configured according to the reference image to be identified. In this embodiment, the preset reference image can be preset by the video playback platform, pre-uploaded by the user, or selected by the user according to the options provided by the video playback platform, etc. Of course, it can also be from other sources, which can be flexibly configured according to actual needs.

[0147] The specific implementation method can refer to the description of step 103 in the above-mentioned image recognition method embodiment, which will not be repeated here.

[0148] 206. The terminal detects whether the feature point matches the reference point correctly.

[0149] The terminal can use the GMS algorithm to detect whether each feature point correctly matches the reference point with the shortest Hamming distance. The specific implementation method can refer to the description of step 104 in the above-mentioned image recognition method embodiment, which will not be repeated here.

[0150] 207. If the number of correctly matched feature points in the feature point set exceeds a preset value, the terminal determines that the image matches the reference image successfully.

[0151] The specific implementation method can refer to the description of step 105 in the above-mentioned image recognition method embodiment, which will not be repeated here.

[0152] As can be seen from the above, in the process of video playback, the embodiment of the present invention intercepts the image of the preset area in the video screen to perform image recognition; then, the pixel points in the image are identified with feature points to obtain the candidate feature point set corresponding to the image; the feature points in the candidate feature point set are screened according to the preset rules to obtain the feature point set of the image; then, the descriptor of the feature point in the feature point set is obtained, and according to the descriptor of the feature point, the reference point with the shortest Hamming distance to the feature point is obtained in the preset reference image feature set, and the feature point and the reference point are detected to be correctly matched to eliminate the wrong match and ensure the accuracy of image recognition; if the number of correctly matched feature points in the feature point set exceeds the preset value, it can be considered that the match is successful and the recognition of the reference image is achieved. This solution reduces the amount of calculation in the subsequent feature point matching by screening the feature points in the image, and uses the high-quality feature points obtained by screening for matching, thereby improving the accuracy of image recognition. Therefore, this solution effectively improves the efficiency of image recognition in the video. When the user is watching the video, the terminal can recognize the specified image in the video in real time to prompt the user and improve the user's viewing experience.

[0153] According to the method described in the previous embodiment, the image recognition device is specifically integrated into a server of a video playback platform as an example for further detailed description.

[0154] In one embodiment, an image recognition method is provided. The difference from the embodiment of the above image recognition method is that: in this embodiment, the server can configure a reference image feature set according to the reference image by itself for image recognition, and can preliminarily screen the images in a preset area to improve the recognition efficiency.

[0155] Referring to Figure 3a , the method includes:

[0156] 301. The server obtains a preset reference image and its corresponding preset area.

[0157] Users such as viewers or maintenance personnel of the video playback platform can upload pictures as custom preset reference images for the server to recognize this reference image during video playback. And after the user uploads the reference image, the area corresponding to this reference image needs to be input correspondingly, so that the server can recognize the images in the specified area in the video picture and improve the recognition efficiency.

[0158] The server obtains the picture uploaded by the user as the preset reference image, and takes the area input by the user as the preset area corresponding to the reference image, so as to facilitate subsequent recognition of the images in the preset area of the video picture and determine whether it matches the reference image.

[0159] Of course, in some embodiments, the user can also directly upload a picture containing the reference image, mark the area where the reference image is located in the picture, then the server can intercept the image according to the area marked by the user, configure the intercepted image as the preset reference image, and configure the preset area according to the area position marked by the user.

[0160] For example, referring to Figure 3b , taking game live broadcast as an example, the user can upload a game screenshot in advance, and then use a marking box in the screenshot to mark the area where the reference image "Peerless" is located. Then the server intercepts the image of the word "Peerless" as the preset reference image according to the marking box, and configures the area marked by the marking box as the preset area.

[0161] 302. The server performs feature point recognition on the reference image to obtain a reference image feature set.

[0162] After obtaining the reference image, the server performs feature point recognition on the reference image, and configures the recognized feature points as reference points into a set to obtain a reference image feature set.

[0163] In some embodiments, step 302 may include: the server takes the pixel points in the reference image as the centers, and determines the corresponding reference points for the pixel points according to a preset radius; obtains the gray value information of the pixels and their corresponding reference points, and determines whether the pixel points are feature points according to the gray value information; if the pixel points are feature points, then takes the pixel points as reference points and configures them into the reference image feature set.

[0164] Thus, a feature point set corresponding to the reference image is obtained, which includes each reference point for feature point matching.

[0165] Of course, in some embodiments, the feature points of the reference image can also be screened. Step 302 may include: the server identifies the pixel points in the reference image to obtain a candidate feature point set corresponding to the reference image; screens the feature points in the candidate feature point set according to a preset rule to obtain the reference image feature set.

[0166] For example, referring to Figure 3c , the feature points identified for the reference image of the word "peerless" are represented by white dots in the figure.

[0167] Then, the server obtains the descriptors of each reference point in the reference image feature set for calculating the Hamming distance.

[0168] 303. During the video playback, the server intercepts the image of a preset area in the video frame.

[0169] During the video playback, the server intercepts the image of a preset area in the video frame according to a preset frequency to detect whether it matches the reference image. For example, referring to Figure 3d , the server takes a screenshot in the video frame as shown in Figure 3d .

[0170] 304. The server obtains the hash string of the image and calculates the Hamming distance between the hash string and the preset reference image hash string.

[0171] The server can use the mean hash algorithm or phash (Perceptual hash algorithm), that is, the perceptual hash algorithm, etc., to obtain the hash string of the image to realize the preliminary detection of the intercepted image.

[0172] In this embodiment, taking the server using phash to obtain the hash string of the image as an example, the server first reduces the size of the image, such as 32×32, to simplify the calculation. Then, the server converts the image into a grayscale image to simplify the color and further reduce the calculation amount.

[0173] The server performs DCT (Discrete Cosine Transform) on the reduced grayscale image to obtain a 32×32 DCT coefficient matrix. Then, the server only retains the 8×8 DCT matrix in the upper left corner of the DCT coefficient matrix, which represents the lowest frequency in the image. In addition, the server calculates the mean of the DCT.

[0174] Then, based on the 8×8 DCT matrix, the server sets the return value of the values ​​in the matrix that are greater than or equal to the DCT mean to "1" and the return value of the values ​​that are less than the DCT mean to "0", and obtains the return value of each value in the matrix. Then, the server combines the return values ​​of each value in the matrix in a preset order to form a 64-bit integer. This 64-bit integer is the hash string of the image, that is, the fingerprint of the image.

[0175] Then, the server obtains the Hamming distance between the image hash string and the preset reference image hash string according to the image hash string. The preset reference image hash string can be obtained by the server according to the reference image, for example, using a mean hash algorithm or phash.

[0176] 305. If the Hamming distance is less than a preset first threshold, the server obtains a feature point set of the image and descriptors of the feature points in the feature point set.

[0177] If the Hamming distance between the hash string of the image and the preset reference image hash string is less than a preset first threshold, it can be preliminarily determined that the image has a high similarity with the reference image and the probability of successful matching between the image and the reference image is high.

[0178] The server obtains the feature point set of the image and the descriptors of the feature points in the feature point set to further identify the image. The specific implementation method can refer to the description of step 102 or 202-204 in the above-mentioned image recognition method embodiment, which will not be repeated here.

[0179] 306. The server obtains a reference point having the shortest Hamming distance to the feature point from a preset reference image feature set according to the descriptor of the feature point.

[0180] After obtaining the descriptors of the feature points in the feature point set, the server quickly obtains the reference point with the shortest Hamming distance to each feature point in the preset reference image feature point set through brute force enumeration.

[0181] 307. The server detects whether the feature point and the reference point are correctly matched.

[0182] The server can use the GMS algorithm to detect whether each feature point is correctly matched with the reference point with the shortest Hamming distance. For the specific implementation method, reference can be made to the description in step 104 of the above-mentioned image recognition method embodiment, which will not be elaborated here.

[0183] For example, referring to Figure 3e , the image of the word "Peerless" on the left is the image intercepted by the server in the video frame, and the image of the word "Peerless" on the right is the reference image.

[0184] The server obtains the neighborhood point set of the feature points in the image of the word "Peerless" as the first neighborhood point set, and obtains the neighborhood point set of the reference point with the shortest Hamming distance from this feature point in the reference image as the second neighborhood point set; then, it matches the pixel points in the first neighborhood point set and the second neighborhood point set, and obtains the number of pairs of successfully matched pixel points. As Figure 3e shown, the server connects the feature points of the intercepted image in the video frame with the reference point with the shortest Hamming distance. If the neighborhood pixel points of the feature point are successfully matched with the neighborhood pixel points of the reference point, the successfully matched pixel point pairs are connected with lines.

[0185] If the number of pairs of successfully matched pixel points exceeds a preset second threshold, it is determined that the feature point is correctly matched with the reference point with the shortest Hamming distance.

[0186] 308. If the number of correctly matched feature points in the feature point set exceeds a preset value, the server determines that the image matches the reference image successfully.

[0187] If the number of correctly matched feature points in the feature point set exceeds a preset value, the server determines that the similarity between the image intercepted in the video frame and the reference image is extremely high and the match is successful.

[0188] In some embodiments, after the server determines that the image in the preset area of the video frame matches the reference image successfully, it can determine that a reference event corresponding to the reference image has occurred in the current video. Then, it can display a prompt message corresponding to the reference event in the video frame and can also be assisted by a preset prompt sound.

[0189] For example, the server determines that the image of the word "Peerless" in the preset area of the video frame as shown in Figure 3d matches the reference image of the word "Peerless" successfully, and the "Peerless" event has occurred. Then the server can obtain the preset prompt message corresponding to the Peerless event and can display the preset prompt message in the video frame as shown in Figure 3d .

[0190] Then, the server continues to execute step 303 to intercept the image in the preset area of the video frame for image recognition.

[0191] 309. If the Hamming distance exceeds a preset first threshold, the server determines that the image fails to match the reference image.

[0192] If the Hamming distance between the hash string of the captured image in the video screen and the preset reference image hash string exceeds the preset first threshold, it is determined that the image has a low similarity with the reference image and the matching fails, and there is no need to further obtain feature points for matching.

[0193] In some embodiments, if the number of correctly matched feature points in the feature point set does not exceed a preset value, the server can determine that the image of the preset area in the video screen fails to match the reference image, and continue to execute step 303 to capture the image of the preset area in the video screen for image recognition.

[0194] As can be seen from the above, the embodiment of the present invention obtains a preset reference image and its corresponding preset area, identifies the feature points of the reference image, and obtains a reference image feature set; then, during the video playback process, the image of the preset area in the video screen is intercepted and image recognition is performed, thereby realizing the customization of the reference image and improving the flexibility of image recognition. Then, by obtaining the hash string of the image and comparing it with the preset reference image hash string to obtain the Hamming distance, a preliminary comparison of the image fingerprint is achieved. If this Hamming distance is less than the preset first threshold, the feature point set of the image and the descriptors of each feature point in the feature point set are further obtained to match the feature points. Therefore, when the reference image to be identified is updated, the scheme only needs to update one reference image to achieve rapid update iteration. Compared with the existing mainstream scheme that requires a lot of manpower to collect materials and retrain, the embodiment of the present invention can effectively save the manpower cost of operation and maintenance. In addition, the solution can also compare the fingerprint of the image by obtaining the hash string of the image to achieve preliminary recognition and screening of the image. For images with high similarity, feature point matching is further performed to ensure the accuracy of image recognition. For images with low similarity, feature point matching is no longer required, saving unnecessary feature point matching time and improving image recognition speed. As a result, the solution saves a lot of cloud computing resources, supports rapid iterative updates, and saves operation and maintenance costs.

[0195] In order to better implement the above method, an embodiment of the present invention further provides an image recognition device, which can be integrated into a network device such as a terminal or a server.

[0196] For example, Figure 4 As shown, the distributed denial of service attack protection device may include a screenshot unit 401, a feature unit 402, a matching unit 403, a detection unit 404 and a determination unit 405, as follows:

[0197] (i) Screen capture unit 401;

[0198] The screenshot unit 401 is used to capture an image of a preset area in the video screen during video playback.

[0199] For example, when broadcasting live, or playing movies, TV and other videos, the image recognition device integrated in the playback background can perform image recognition on the video. Of course, when watching live broadcasts, or playing movies, TV and other videos, the image recognition device integrated in the client can also perform image recognition on the video. When a specified event that needs to be identified occurs in the video screen, such as exciting game events such as "penta kill" or "quadruple kill", the corresponding image will be displayed, such as the words "penta kill" or "quadruple kill", etc. Therefore, these images with the words "penta kill" or "quadruple kill" can be used as reference images to perform image recognition in the video screen. If the reference image is recognized to appear in the video screen, it can be determined that the specified event has occurred.

[0200] During video playback, the screenshot unit 401 can capture an image of a preset area in the video screen according to a preset frequency to perform image recognition. The preset area can be configured according to the position or area displayed by the reference image to be detected. In addition, the preset area can be configured based on experience of the position displayed by the reference image, or can be configured based on a fixed position displayed by the reference image, and can be flexibly configured or adjusted.

[0201] After obtaining the image of the preset area, the screenshot unit 401 can pre-process the image, including: scaling the image to a preset size and / or grayscale processing the image, etc. The execution order of image scaling and grayscale processing is not limited and can be flexibly configured according to actual needs. Scaling the image to a preset size can standardize the image to avoid the difference caused by different image sizes affecting the recognition result; grayscale processing the image, the pixels in the image can be represented by grayscale values, and the description of the grayscale image still reflects the distribution and characteristics of the overall and local chromaticity and brightness levels of the entire image, just like the color image, but it can reduce the amount of calculation for subsequent image recognition relative to the color image.

[0202] For example, the preset frequency may be 1 time / second, and the capture unit 401 captures an image of a preset area in the video screen every second for detection.

[0203] If multiple reference images need to be identified during video playback, the capture unit 401 captures images of preset areas corresponding to the reference images respectively to perform image recognition.

[0204] (ii) feature unit 402;

[0205] The feature unit 402 is used to obtain a feature point set of the image and descriptors of the feature points in the feature point set.

[0206] After obtaining the image of the preset area, the feature unit 402 identifies feature points of the image and configures the identified feature points into a set to obtain a feature point set of the image. Then, each feature point in the feature point set is described to obtain a descriptor of each feature point.

[0207] As an implementation, the feature unit 402 may include an identification subunit and a description subunit. The identification subunit is used to determine the reference point corresponding to the pixel point according to a preset radius with the pixel point in the image as the center of the circle; obtain the gray value information of the pixel and its corresponding reference point, and determine whether the pixel point is a feature point according to the gray value information; if the pixel point is a feature point, the pixel point is configured into the feature point set of the image. The description subunit is used to obtain the descriptor of the feature point.

[0208] The recognition subunit can detect whether each pixel in the image is a feature point. Specifically, the image recognition device takes a pixel in the image, takes the pixel as the center, draws a circle on the image according to a preset radius, and takes the pixel passing through the circle as the reference point corresponding to the pixel.

[0209] Then, the recognition subunit obtains the gray value information of this pixel and its corresponding reference point. If the gray values ​​of n consecutive reference points are greater than the reference point, or the gray values ​​of n consecutive reference points are less than the reference point, then this pixel can be determined as a feature point. Where n is a preset value, which can be flexibly configured according to actual needs.

[0210] Of course, in order to ensure the accuracy of the feature points, a threshold can also be set to filter out reference points that are not much different from the pixel points. For example, the grayscale value of this pixel point is I, and the threshold is t. If there are more than n consecutive pixel points on the circumference whose grayscale values ​​are greater than I+t, or smaller than It, the recognition subunit determines that this pixel point is a feature point, otherwise it is not.

[0211] Then, the recognition subunit configures the feature point into the feature set of the image. Similarly, the recognition subunit detects whether each pixel in the image is a feature point, and configures all the feature points into the feature set of the image, thereby obtaining the feature set of the image.

[0212] For example, refer to Figure 1c , the recognition subunit detects whether pixel 0 in the image is a feature point, taking the preset radius as 3 pixels and n as 12.

[0213] Then, the recognition subunit draws a circle with pixel 0 as the center and 3 pixels as the radius. The circumference of the circle passes through 16 pixels in total. The recognition subunit uses these 16 pixels as reference points corresponding to pixel 0, and numbers these 16 reference points in chronological order. First, the recognition subunit can detect the grayscale values ​​of reference points 1, 5, 9, and 13. If three or more of the grayscale values ​​of these four reference points are greater than or less than the grayscale value of pixel 0, the recognition subunit further detects whether there are 12 consecutive pixel values ​​greater than the pixel value of pixel 0, or whether there are 12 consecutive pixel values ​​less than the pixel value of pixel 0. If so, the recognition subunit determines that pixel 0 is a feature point and configures it into the feature point set of the image.

[0214] If there are not three or more grayscale values ​​of reference points 1, 5, 9 and 13 that are greater than or less than the grayscale value of pixel 0, then pixel 0 is determined not to be a feature point. This achieves a preliminary judgment of the feature point, reduces the amount of data calculation, and improves the judgment speed.

[0215] After obtaining the feature set of the image, the description subunit needs to obtain the descriptor of each feature point in the feature set. As an implementation method, the description subunit can use the BRIEF (Binary Robust Independent Elementary Features) algorithm to obtain the descriptor of each feature. Specifically, the description subunit selects m pairs of pixel points pi and qi (i=1, 2, ..., m) in the neighborhood of a feature point according to a preset mode, and the grayscale value of Pi is I(pi), and the grayscale value of qi is I(qi). Then, the description subunit compares the grayscale values ​​of each pair of pixels. If I(pi)>I(qi), the description subunit generates a return value of 1, otherwise the return value is 0. When all pixel pairs are compared, the description subunit configures the obtained return value as a binary string of length m according to a preset order as the descriptor of this feature point. By analogy, the description subunit obtains the descriptors of each feature point in the feature point set respectively.

[0216] (iii) matching unit 403;

[0217] The matching unit 403 is used to obtain a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set according to the descriptor of the feature point.

[0218] The preset reference image feature set includes reference points for feature point matching, wherein each reference point is characterized by a descriptor. It should be noted that the preset reference image feature set is configured in advance according to the reference image to be identified as required, wherein each reference point is a feature point in the reference image. Furthermore, the length of the descriptor string of each reference point is consistent with the length of the feature point descriptor string obtained by the feature unit 402.

[0219] The Hamming distance between two strings of equal length refers to the number of different characters in the corresponding positions of the two strings. In other words, it is the number of characters that need to be replaced to transform one string into another string. The matching unit 403 obtains the Hamming distance with each reference point in the reference image feature set by brute force enumeration according to the descriptors of each feature point of the image, and then selects the reference point with the shortest Hamming distance. Among them, the reference point with the shortest Hamming distance with the feature point is also the reference point that is most similar to the feature point.

[0220] Thus, the matching unit 403 obtains the reference points with the shortest Hamming distances of the feature points.

[0221] (iv) detection unit 404;

[0222] The detection unit 404 is used to detect whether the feature points and the reference points are correctly matched.

[0223] In order to eliminate false matches, the detection unit 404 may also detect whether the feature point and the reference point with the shortest Hamming distance therebetween are correctly matched.

[0224] As an implementation mode, the detection unit 404 is specifically used to: obtain the neighborhood point set of the feature point as the first neighborhood point set, and obtain the neighborhood point set of the reference point as the second neighborhood point set; match the pixel points in the first neighborhood point set and the second neighborhood point set, and obtain the number of successfully matched pixel point pairs; if the number of successfully matched pixel point pairs exceeds a preset second threshold, it is determined that the feature point and the reference point are correctly matched.

[0225] If the feature point is correctly matched with the reference point with the shortest Hamming distance, the pixels around the two should also be the same. Therefore, the matching of the neighborhood pixels can be used to detect whether the two are correctly matched. For example, GMS (Grid-based Motion Statistics) can be used to detect whether the feature point is correctly matched with the reference point with the shortest Hamming distance.

[0226] The detection unit 404 can extract the neighborhood pixel points of the feature point according to a preset size, such as 3×3, configure the neighborhood point set of the feature point, and use the neighborhood point set of the feature point as the first neighborhood point set. In addition, the detection unit 404 obtains the neighborhood pixel points of the reference point according to the preset size, configures the neighborhood point set of the reference point, and uses the neighborhood point set of the reference point as the second neighborhood point set. It should be noted that when extracting neighborhood pixel points, the positional relationship between the feature point and its neighborhood pixel points is the same as the positional relationship between the reference point and its neighborhood pixel points. Therefore, the pixels in the first neighborhood point set and the second neighborhood point set are positionally corresponding. For example, the feature point can be used as the center point, and the neighborhood pixel points around it can be extracted according to the preset size to obtain the first neighborhood point set. Similarly, the reference point can be used as the center point, and the neighborhood pixel points around it can be extracted according to the preset size to obtain the second neighborhood point set.

[0227] The detection unit 404 may extract the neighborhood pixel points of the feature point in the intercepted image, and extract the neighborhood pixel points of the reference point in the reference image. Alternatively, the detection unit 404 may extract the neighborhood pixel points of the feature point in the feature point set, and extract the neighborhood pixel points of the reference point in the reference image feature set, so as to improve the accuracy of matching. The specific configuration may be flexible according to actual needs.

[0228] After obtaining the first neighborhood point set and the second neighborhood point set, the detection unit 404 detects whether the pixels at the same position in the first neighborhood point set and the second neighborhood point set are the same. Specifically, the pixels at the same position in the first neighborhood point set and the second neighborhood point set are taken as pixel pairs, descriptors of the two pixels in the pixel pairs are obtained, and the Hamming distance of the two pixels in the pixel pairs is obtained according to the descriptors. If the Hamming distance is less than a preset value, it is determined that the pixel pair is matched successfully.

[0229] The detection unit 404 counts the number of successfully matched pixel pairs. If the number of successfully matched pixel pairs exceeds the preset second threshold, it means that the feature point and the surrounding pixels of the reference point with the shortest Hamming distance are substantially the same, and the detection unit 404 can determine that the feature point is correctly matched with the reference point with the shortest Hamming distance.

[0230] Thus, the detection unit 404 determines whether each feature point in the feature point set is correctly matched with the reference point with the shortest Hamming distance thereto, and obtains a detection result.

[0231] (V) determining unit 405;

[0232] The determination unit 405 is configured to determine that the image matches the reference image successfully if the number of correctly matched feature points in the feature point set exceeds a preset value.

[0233] The determination unit 405 counts the number of feature points in the feature point set that are correctly matched with the reference point with the shortest Hamming distance.

[0234] Since the reference image corresponds to the reference image feature set, each reference point in the reference image feature set is extracted from the reference image. Therefore, if the number of correctly matched feature points exceeds the preset value, the determination unit 405 can determine that the image in the preset area in the video screen is highly similar to the reference image, and the match is successful, thereby realizing the recognition of the reference image.

[0235] In some embodiments, after determining that the image of the preset area in the video screen matches the reference image successfully, the determination unit 405 can determine that a reference event corresponding to the reference image has occurred in the current video, and then a prompt message corresponding to the reference event can be displayed in the video screen, and a preset prompt sound can be provided. Then, the screenshot unit 401 continues to capture the image of the preset area in the video screen for image recognition.

[0236] In some embodiments, if the number of correctly matched feature points in the feature point set does not exceed a preset value, it can be determined that the image of the preset area in the video screen fails to match the reference image, and the screenshot unit 401 continues to capture the image of the preset area in the video screen for image recognition.

[0237] As can be seen from the above, in the process of video playback, the screenshot unit 401 captures the image of the preset area in the video screen in the embodiment of the present invention to perform image recognition; then, the feature unit 402 obtains the feature point set of the image and the descriptor of the feature point in the feature point set; then, the matching unit 403 obtains the reference point with the shortest Hamming distance to the feature point in the preset reference image feature set according to the descriptor of the feature point, and the detection unit 404 detects whether the feature point and the reference point are correctly matched to eliminate wrong matching and ensure the accuracy of image recognition; if the number of correctly matched feature points in the feature point set exceeds the preset value, the determination unit 405 can consider that the image matches the reference image, and the recognition of the reference image is realized. This solution does not need to use a complex training model. By matching the feature points of the image in a specific area with the reference points in the reference image feature set, it is determined whether the image in a specific area is a reference image, which reduces the amount of data calculation, speeds up the image recognition processing speed, and realizes the rapid recognition of the specified image. Moreover, after obtaining the reference point with the shortest Hamming distance between the feature point and the reference point, the solution can effectively eliminate false matches and improve the accuracy of image recognition by correctly matching the feature point with the reference point with the shortest Hamming distance. In addition, the solution can perform image recognition on the video in real time in the background of the video platform or in the user terminal. The increase in the amount of video playback on the video platform will not affect the accuracy and speed of image recognition. The application is more flexible and convenient, saving a lot of cloud resources and operation and maintenance costs. Therefore, compared with the existing technology, the solution occupies less resources, reduces the burden of data processing, and ensures the efficiency of image recognition, and can realize real-time image recognition of video images.

[0238] In one embodiment, an image recognition device is provided, which is different from the above-mentioned image recognition device embodiment in that the image recognition device in this embodiment screens the feature points in the feature point set, which can further reduce the amount of data calculation and improve the accuracy of image recognition. Figure 5 As shown, the device includes a screenshot unit 501, a feature unit 502, a matching unit 503, a detection unit 504 and a determination unit 505, wherein the feature unit 502 includes a candidate subunit 5021, a screening subunit 5022 and a description subunit 5023, as follows:

[0239] (i) Screen capture unit 501;

[0240] The screenshot unit 501 is used to capture an image of a preset area in the video screen during video playback.

[0241] For example, when a user uses a terminal to watch a live video, the screenshot unit 501 can capture an image of a preset area in the video screen at preset time intervals during the video playback for image recognition.

[0242] (ii), candidate subunit 5021;

[0243] The candidate subunit 5021 is used to perform feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image.

[0244] The candidate subunit 5021 identifies feature points in the image and configures a set of candidate feature points corresponding to the image.

[0245] As an implementation method, the candidate subunit 5021 can be specifically used to: take the pixel point in the image as the center of the circle, and determine the reference point corresponding to the pixel point according to a preset radius; obtain the grayscale value information of the pixel and its corresponding reference point, and determine whether the pixel point is a feature point according to the grayscale value information; if the pixel point is a feature point, then configure the pixel point into the candidate feature point set of the image.

[0246] The specific implementation method can refer to the description of the feature unit 402 in the above-mentioned image recognition device embodiment, which will not be repeated here.

[0247] (iii) screening subunit 5022;

[0248] The screening subunit 5022 is used to screen the feature points in the candidate feature point set according to a preset rule to obtain a feature point set of the image.

[0249] After obtaining the candidate feature point set, the screening subunit 5022 screens out some feature points from the candidate feature point set according to a preset rule, and configures the screened feature points into the feature point set of the image.

[0250] Among them, the preset rules can be to randomly screen a preset number of feature points, or to screen out feature points at preset positions in the image. Of course, it can also be a rule to screen out feature points whose corner point response values ​​exceed preset values, so that the screened feature points are more representative, which can not only reduce the calculation amount of feature point matching, but also improve the accuracy of image recognition.

[0251] In some embodiments, the preset rule is to filter out a preset number of corner points, and the screening subunit 5022 can be specifically used to: calculate the corner point response value of each feature point in the candidate feature point set respectively; based on the corner point response value, filter out a preset number of feature points in the candidate feature point set, and configure a feature point set of the image.

[0252] The preset number can be flexibly configured according to actual needs, such as 200 or 600. In this embodiment, the preset number is x.

[0253] In some embodiments, the screening subunit 5022 may use the Harris algorithm to calculate the corner point response value of each feature point in the candidate feature point set.

[0254] Then, the screening subunit 5022 sorts the feature points in the candidate feature point set according to the corner point response value to obtain a feature point sequence whose corner point response values ​​are arranged from large to small. The screening subunit 5022 selects the feature points ranked from the 1st to the xth position in the feature point sequence as the screened corner points. The screening subunit 5022 configures these x feature points into the feature point set, that is, obtains the feature point set of the image.

[0255] Therefore, the response values ​​of the corner points of the feature points in the screened feature point set are relatively large and the features are obvious, which can improve the accuracy of feature point matching.

[0256] In some embodiments, the number of feature points in the feature point set can be configured according to the nature of the image to improve the flexibility and accuracy of feature point screening. Specifically, the feature unit 502 can also include a quantity subunit (not shown in the figure), which is used to determine the required quantity of feature points according to the position of the preset area and configure the required quantity to a preset quantity.

[0257] When performing image recognition, preset areas at different positions correspond to different images that need to be recognized. For example, images such as text and hero portraits in games have different corresponding preset area positions.

[0258] Therefore, the quantity subunit can determine the corresponding reference image to be recognized according to the preset area position of the captured image, and further determine the required quantity of feature points when performing image recognition according to the category of the reference image.

[0259] For example, if the reference image corresponding to the preset area is text and its category is text, and its image edge is relatively obvious, the demand for feature points can be relatively small during image recognition, which not only reduces the amount of feature point matching calculations, but also ensures the accuracy of image recognition.

[0260] If the reference image corresponding to the preset area is a game hero portrait, and the category is hero, the edge of the image may not be so obvious. In this case, when performing image recognition, the demand for feature points can be relatively large, which slightly reduces the amount of feature point matching calculations and ensures the accuracy of image recognition.

[0261] As a result, it is possible to filter a specific number of feature points for matching according to the category of the image, thereby improving the flexibility of image recognition.

[0262] (iv), description subunit 5023;

[0263] The description subunit may use the BRIEF algorithm to obtain the descriptor of each feature. The specific implementation method may refer to the description of the feature unit 402 in the above-mentioned image recognition device embodiment, and will not be repeated here.

[0264] (V), matching unit 503;

[0265] The matching unit 503 is used to obtain a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set according to the descriptor of the feature point.

[0266] The matching unit 503 finds the reference point with the shortest Hamming distance to each characteristic point in the feature point set in the preset reference image set through brute force enumeration, thereby obtaining the reference point most similar to each characteristic point.

[0267] The preset reference image feature set is configured according to the reference image to be identified. In this embodiment, the preset reference image can be preset by the video playback platform, pre-uploaded by the user, or selected by the user according to the options provided by the video playback platform, etc. Of course, it can also be from other sources, which can be flexibly configured according to actual needs.

[0268] The specific implementation method can refer to the description of the matching unit 403 in the above-mentioned image recognition device embodiment, which will not be repeated here.

[0269] (vi), detection unit 504;

[0270] The detection unit 504 can use the GMS algorithm to detect whether each feature point correctly matches the reference point with the shortest Hamming distance. The specific implementation method can refer to the description of the detection unit 404 in the above-mentioned image recognition device embodiment, which will not be repeated here.

[0271] (VII) Determine unit 505.

[0272] The specific implementation manner may refer to the description of the determination unit 405 in the above-mentioned image recognition device embodiment, which will not be described in detail here.

[0273] As can be seen from the above, in the process of video playback, the screenshot unit 501 captures the image of the preset area in the video screen in the embodiment of the present invention to perform image recognition; then, the candidate subunit 5021 performs feature point recognition on the pixel points in the image to obtain the candidate feature point set corresponding to the image; the screening subunit 5022 screens the feature points in the candidate feature point set according to the preset rules to obtain the feature point set of the image; then, the description subunit 5023 obtains the descriptor of the feature point in the feature point set, so that the matching unit 503 can obtain the reference point with the shortest Hamming distance to the feature point in the preset reference image feature set according to the descriptor of the feature point; the detection unit 504 detects whether the feature point and the reference point are correctly matched to eliminate wrong matching and ensure the accuracy of image recognition; if the number of correctly matched feature points in the feature point set exceeds the preset value, the determination unit 505 can consider that the current matching is successful and the recognition of the reference image is realized. This scheme reduces the amount of calculation when the feature point matching is performed subsequently by screening the feature points in the image, and uses the high-quality feature points obtained by screening for matching, thereby improving the accuracy of image recognition. Therefore, the solution effectively improves the efficiency of image recognition in videos. When users watch videos, the terminal can recognize the specified image in the video in real time to prompt the user and improve the user's viewing experience.

[0274] In one embodiment, an image recognition device is provided, which is different from the above-mentioned image recognition device embodiment in that the image recognition device in this embodiment can configure a reference image feature set according to a reference image to perform image recognition, and can perform preliminary identification of images in a preset area to improve recognition efficiency. Figure 6 As shown, the device includes a reference unit 601, a screenshot unit 602, a hash unit 603, a feature unit 604, a matching unit 605, a detection unit 606 and a determination unit 607:

[0275] (i) reference unit 601;

[0276] The reference unit 601 is used to obtain a preset reference image and its corresponding preset area; perform feature point recognition on the reference image to obtain a reference image feature set.

[0277] Users such as viewers or maintenance personnel of the video playback platform can upload pictures as customized preset reference images for the video playback device to identify the reference images during video playback. In addition, after uploading the reference image, the user needs to input the area corresponding to the reference image so that the server can recognize the image of the specified area in the video screen, thereby improving recognition efficiency.

[0278] The reference unit 601 obtains the picture uploaded by the user as a preset reference image, and uses the area input by the user as the preset area corresponding to the reference image, so as to facilitate subsequent recognition of the image in the preset area of ​​the video screen and determine whether it matches the reference image.

[0279] Of course, in some embodiments, the user can also directly upload a picture containing a benchmark image, and identify the area where the benchmark image is located in the picture. The benchmark unit 601 can then capture the image based on the area marked by the user, configure the captured image as a preset benchmark image, and configure the preset area based on the location of the area marked by the user.

[0280] After obtaining the reference image, the reference unit 601 performs feature point recognition on the reference image, and configures the recognized feature points as reference points into a set to obtain a reference image feature set.

[0281] In some embodiments, the reference unit 601 can be specifically used to: use the pixel point in the reference image as the center of the circle, and determine the reference point corresponding to the pixel point according to a preset radius; obtain the grayscale value information of the pixel and its corresponding reference point, and determine whether the pixel point is a feature point according to the grayscale value information; if the pixel point is a feature point, then use the pixel point as the reference point and configure it into the reference image feature set.

[0282] Thus, a feature point set corresponding to the reference image is obtained, which includes each reference point used for feature point matching.

[0283] Of course, in some embodiments, the feature points of the reference image can also be screened, and the reference unit 601 can be specifically used to: identify the pixel points in the reference image to obtain a set of candidate feature points corresponding to the reference image; and screen the feature points in the candidate feature point set according to preset rules to obtain a reference image feature set.

[0284] (ii) screenshot unit 602;

[0285] The capture unit 602 is used to capture an image of a preset area in the video screen during video playback.

[0286] During video playback, the capture unit 602 captures an image of a preset area in the video screen according to a preset frequency to detect whether it matches the reference image.

[0287] (iii) Hash unit 603;

[0288] The hash unit 603 is used to obtain a hash string of the image, and calculate the Hamming distance between the hash string and a preset reference image hash string; if the Hamming distance exceeds a preset first threshold, it is determined that the image fails to match the reference image.

[0289] The hash unit 603 may use a mean hash algorithm or a phash (Perceptual hash algorithm), that is, a perceptual hash algorithm, etc. to obtain a hash string of the image and implement preliminary detection of the captured image.

[0290] In this embodiment, the hash unit 603 uses phash to obtain the hash string of the image. The hash unit 603 first reduces the size of the image, such as 32×32, to simplify the calculation. Then, the hash unit 603 converts the image into a grayscale image to simplify the color and further simplify the calculation amount.

[0291] The hash unit 603 performs DCT (Discrete Cosine Transform) on the reduced grayscale image to obtain a 32×32 DCT coefficient matrix. Then, the hash unit 603 only retains the 8×8 DCT matrix at the upper left corner of the DCT coefficient matrix, which represents the lowest frequency in the image. In addition, the hash unit 603 calculates the mean of the DCT.

[0292] Then, according to the 8×8 DCT matrix, the hash unit 603 sets the return value of the value in the matrix that is greater than or equal to the DCT mean value to "1" and the return value of the value that is less than the DCT mean value to "0" to obtain the return value of each value in the matrix. Then, the hash unit 603 combines the return values ​​of each value in the matrix in a preset order to form a 64-bit integer. This 64-bit integer is the hash string of the image, that is, the fingerprint of the image.

[0293] Then, the hash unit 603 obtains the Hamming distance between the image hash string and the preset reference image hash string. The preset reference image hash string can be obtained by the hash unit 603 according to the reference image, for example, using a mean hash algorithm or phash.

[0294] If the Hamming distance between the hash string of the captured image in the video screen and the preset reference image hash string exceeds the preset first threshold, the hash unit 603 determines that the image and the reference image have low similarity and the matching fails, and there is no need to further obtain feature points for matching.

[0295] (iv), feature unit 604;

[0296] The feature unit 604 is used to obtain a feature point set of the image and a descriptor of the feature point in the feature point set if the Hamming distance is less than a preset first threshold.

[0297] If the Hamming distance between the hash string of the image and the preset reference image hash string is less than a preset first threshold, it can be preliminarily determined that the image has a high similarity with the reference image and the probability of successful matching between the image and the reference image is high.

[0298] The feature unit 604 obtains the feature point set of the image and the descriptors of the feature points in the feature point set to further identify the image. The specific implementation method can refer to the description of the feature unit 402 in the above-mentioned image recognition device embodiment, or the description of the candidate subunit 5021, the screening subunit 5022 and the description subunit 5023 in the feature unit 502, which will not be repeated here.

[0299] (V), matching unit 605;

[0300] After obtaining the descriptors of the feature points in the feature point set, the matching unit 605 quickly obtains the reference point with the shortest Hamming distance to each feature point in the preset reference image feature point set through brute force enumeration.

[0301] (vi), detection unit 606;

[0302] The detection unit 606 may use the GMS algorithm to detect whether each feature point correctly matches the reference point with the shortest Hamming distance thereto. The specific implementation method may refer to the description of the detection unit 404 in the above-mentioned image recognition device embodiment, which will not be repeated here.

[0303] (VII) Determine unit 607.

[0304] If the number of correctly matched feature points in the feature point set exceeds a preset value, the determination unit 607 determines that the image captured in the video screen has a very high similarity with the reference image and the match is successful.

[0305] As can be seen from the above, the embodiment of the present invention obtains a preset reference image and its corresponding preset area through the reference unit 601, identifies feature points of the reference image, and obtains a reference image feature set; then, during the video playback process, the screenshot unit 602 captures the image of the preset area in the video screen and performs image recognition, thereby realizing the customization of the reference image and improving the flexibility of image recognition. Then, the hash unit 603 obtains the image hash string and compares it with the preset reference image hash string to obtain the Hamming distance, thereby realizing a preliminary comparison of the image fingerprint. If this Hamming distance is less than the preset first threshold, the feature unit 604 obtains the image feature point set and the descriptor of each feature point in the feature point set, and the matching unit 605 matches the feature points. Therefore, when the reference image to be identified is updated, the scheme only needs to update one reference image to achieve rapid update iteration. Compared with the existing mainstream scheme that requires a lot of manpower to collect materials and retrain, the embodiment of the present invention can effectively save the manpower cost of operation and maintenance. In addition, the solution can also compare the fingerprint of the image by obtaining the hash string of the image to achieve preliminary recognition and screening of the image. For images with high similarity, feature point matching is further performed to ensure the accuracy of image recognition. For images with low similarity, feature point matching is no longer required, saving unnecessary feature point matching time and improving image recognition speed. As a result, the solution saves a lot of cloud computing resources, supports rapid iterative updates, and saves operation and maintenance costs.

[0306] The embodiment of the present invention further provides a server, such as Figure 7 As shown, it shows a schematic diagram of the structure of the server involved in the embodiment of the present invention, specifically:

[0307] The server may include one or more processing core processors 701, one or more computer-readable storage media memories 702, a power supply 703, an input unit 704 and other components. Those skilled in the art will appreciate that Figure 7 The server structure shown in the figure does not constitute a limitation on the server, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0308] in:

[0309] Processor 701 is the control center of the server. It uses various interfaces and lines to connect various parts of the entire server. It performs various functions of the server and processes data by running or executing software programs and / or modules stored in memory 702 and calling data stored in memory 702, thereby monitoring the server as a whole.

[0310] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 can mainly include a program storage area and a data storage area. The program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the server, etc.

[0311] The server also includes a power supply 703 for supplying power to various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0312] The server may further include an input unit 704, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0313] Although not shown, the server may further include a display unit, etc., which will not be described in detail here. Specifically in this embodiment, the processor 701 in the server will load the executable files corresponding to the processes of one or more applications into the memory 702 according to the following instructions, and the processor 701 will run the application stored in the memory 702, thereby realizing various functions, as follows:

[0314] During video playback, an image of a preset area in the video screen is captured; a feature point set of the image and descriptors of the feature points in the feature point set are obtained; based on the descriptors of the feature points, a reference point with the shortest Hamming distance to the feature point is obtained in a preset reference image feature set; it is detected whether the feature point and the reference point are correctly matched; if the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully.

[0315] This solution does not require the use of complex training models. By matching the feature points of the image in a specific area with the reference points in the feature set of the reference image, it determines whether the image in a specific area is a reference image, which reduces the amount of data calculation, speeds up the image recognition processing speed, and realizes the rapid recognition of the specified image. In addition, after obtaining the reference point with the shortest Hamming distance between the feature point and its reference point, the solution also performs correct matching detection on the reference point with the shortest Hamming distance between the feature point and its reference point, which can effectively eliminate false matches and improve the accuracy of image recognition. In addition, this solution can perform image recognition on the video in real time in the background server of the video platform. The increase in the amount of video playback in the video platform will not affect the accuracy and speed of image recognition. The application is more flexible and convenient, saving a lot of cloud resources and saving operation and maintenance costs. Therefore, compared with the existing technology, this solution has a small resource footprint, reduces the burden of data processing, and ensures the efficiency of image recognition, and can realize real-time image recognition of video images.

[0316] Accordingly, an embodiment of the present invention further provides a terminal, such as Figure 8 As shown, the terminal may include a radio frequency (RF) circuit 801, a memory 802 including one or more computer-readable storage media, an input unit 803, a display unit 804, a sensor 805, an audio circuit 806, a wireless fidelity (WiFi) module 807, a processor 808 including one or more processing cores, and a power supply 809. Those skilled in the art will appreciate that Figure 8 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0317] The RF circuit 801 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is handed over to one or more processors 808 for processing; in addition, the uplink data is sent to the base station. Generally, the RF circuit 801 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 801 can also communicate with the network and other devices through wireless communication.

[0318] The memory 802 may be used to store software programs and modules. The processor 808 executes various functional applications and data processing by running the software programs and modules stored in the memory 802 .

[0319] The input unit 803 may be used to receive input digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0320] The display unit 804 may be used to display information input by the user or information provided to the user and various graphical user interfaces of the terminal, which may be composed of graphics, text, icons, videos, and any combination thereof. The display unit 804 may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0321] The terminal may also include at least one sensor 805, such as a light sensor, a motion sensor, and other sensors.

[0322] The audio circuit 806 is connected to the speaker, and the microphone can provide an audio interface between the user and the terminal. The audio circuit 806 can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 806 and converted into audio data, and then the audio data is processed by the output processor 808 and sent to another terminal through the RF circuit 801, or the audio data is output to the memory 802 for further processing. The audio circuit 806 may also include an earplug jack to provide communication between an external headset and the terminal.

[0323] WiFi is a short-range wireless transmission technology. The terminal can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 807. It provides users with wireless broadband Internet access. Figure 8 A WiFi module 807 is shown, but it is understandable that it is not an essential component of the terminal and can be omitted as needed without changing the essence of the invention.

[0324] The processor 808 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal and processes data by running or executing software programs and / or modules stored in the memory 802, and calling data stored in the memory 802, so as to monitor the terminal as a whole. Optionally, the processor 808 may include one or more processing cores; preferably, the processor 808 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 808.

[0325] The terminal also includes a power source 809 (such as a battery) for supplying power to various components.

[0326] Although not shown, the terminal may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the processor 808 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 802 according to the following instructions, and the processor 808 will run the applications stored in the memory 802 to implement various functions:

[0327] During video playback, an image of a preset area in the video screen is captured; a feature point set of the image and descriptors of the feature points in the feature point set are obtained; based on the descriptors of the feature points, a reference point with the shortest Hamming distance to the feature point is obtained in a preset reference image feature set; it is detected whether the feature point and the reference point are correctly matched; if the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully.

[0328] This solution does not require the use of complex training models. It determines whether the image in a specific area is a reference image by matching the feature points of the image in a specific area with the reference points in the feature set of the reference image, thereby reducing the amount of data calculation, speeding up the image recognition processing speed, and realizing the rapid recognition of the specified image. In addition, after obtaining the reference point with the shortest Hamming distance between the feature point and the reference point, the solution also performs correct matching detection between the feature point and the reference point with the shortest Hamming distance, which can effectively eliminate false matches and improve the accuracy of image recognition. In addition, the solution can perform image recognition on the video in real time in the user terminal, making the application more flexible and convenient, saving a large amount of cloud resources, and saving operation and maintenance costs. Therefore, compared with the existing technology, the solution occupies less resources, reduces the burden of data processing, and ensures the efficiency of image recognition, and can realize real-time image recognition of video images.

[0329] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0330] To this end, an embodiment of the present invention provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the image recognition methods provided in the embodiment of the present invention. For example, the instructions can execute the following steps:

[0331] During video playback, capturing an image of a preset area in the video screen;

[0332] Acquire a feature point set of the image and descriptors of feature points in the feature point set;

[0333] According to the descriptor of the feature point, obtaining a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set;

[0334] Detecting whether the feature point matches the reference point correctly;

[0335] If the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully.

[0336] This instruction can also perform the following steps:

[0337] Feature point recognition is performed on pixel points in the image to obtain a candidate feature point set corresponding to the image; and feature points in the candidate feature point set are screened according to preset rules to obtain a feature point set of the image.

[0338] This instruction can also perform the following steps:

[0339] The corner point response value of each feature point in the candidate feature point set is calculated respectively; a preset number of feature points are screened out from the candidate feature point set according to the corner point response value, and a feature point set of the image is obtained by configuration.

[0340] This instruction can also perform the following steps:

[0341] The required quantity of feature points is determined according to the position of the preset area, and the required quantity is configured as the preset quantity.

[0342] This instruction can also perform the following steps:

[0343] Taking a pixel point in the image as the center of a circle, determine a reference point corresponding to the pixel point according to a preset radius; obtain grayscale value information of the pixel and its corresponding reference point, and determine whether the pixel point is a feature point according to the grayscale value information; if the pixel point is a feature point, configure the pixel point into a feature point set of the image.

[0344] This instruction can also perform the following steps:

[0345] A hash string of the image is obtained, and a Hamming distance between the hash string and a preset reference image hash string is calculated; if the Hamming distance is less than a preset first threshold, a step of obtaining a feature point set of the image and a descriptor of the feature points in the feature point set is performed; if the Hamming distance exceeds the preset first threshold, it is determined that the image fails to match the reference image.

[0346] This instruction can also perform the following steps:

[0347] The neighborhood point set of the feature point is obtained as a first neighborhood point set, and the neighborhood point set of the reference point is obtained as a second neighborhood point set; pixel points in the first neighborhood point set and the second neighborhood point set are matched, and the number of successfully matched pixel point pairs is obtained; if the number of successfully matched pixel point pairs exceeds a preset second threshold, it is determined that the feature point is correctly matched with the reference point.

[0348] This instruction can also perform the following steps:

[0349] A preset reference image and its corresponding preset area are obtained; feature points of the reference image are identified to obtain a reference image feature set.

[0350] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0351] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0352] Since the instructions stored in the storage medium can execute the steps in any image recognition method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image recognition method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0353] The above is a detailed introduction to an image recognition method, device and storage medium provided in an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image recognition method, characterized in that: include: During video playback, capture an image of a preset area in the video screen; Acquire a feature point set of the image and descriptors of feature points in the feature point set; According to the descriptor of the feature point, obtaining a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set; Detecting whether the feature point matches the reference point correctly; If the number of correctly matched feature points in the feature point set exceeds a preset value, it is determined that the image matches the reference image successfully; The step of obtaining a feature point set of the image comprises: Performing feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image; the set of candidate feature points is obtained by recognition based on gray value information of pixel points in the image and their corresponding reference points; Calculate the corner point response value of each feature point in the candidate feature point set respectively; According to the corner point response values, a preset number of feature points are screened out from the candidate feature point set to obtain a feature point set of the image; The step of selecting a preset number of feature points from the candidate feature point set according to the corner point response values, and configuring the feature point set of the image, further comprises: The corresponding reference image to be identified is determined according to the position of the preset area, the required quantity of feature points is determined according to the category of the identified reference image, and the required quantity is configured as the preset quantity.

2. The method according to claim 1, characterized in that The step of obtaining a feature point set of the image comprises: Taking a pixel point in the image as the center of a circle, determining a reference point corresponding to the pixel point according to a preset radius; Obtaining grayscale value information of the pixel point and its corresponding reference point, and determining whether the pixel point is a feature point according to the grayscale value information; If the pixel point is a feature point, the pixel point is configured into a feature point set of the image.

3. The method according to claim 1, characterized in that The step of obtaining a feature point set of the image and a descriptor of a feature point in the feature point set also includes: Obtaining a hash string of the image, and calculating a Hamming distance between the hash string and a preset reference image hash string; If the Hamming distance is less than a preset first threshold, executing the step of acquiring a feature point set of the image and a descriptor of the feature point in the feature point set; If the Hamming distance exceeds a preset first threshold, it is determined that the image fails to match the reference image.

4. The method according to claim 1, characterized in that The detecting whether the feature point matches the reference point correctly comprises: Acquire a neighborhood point set of the feature point as a first neighborhood point set, and acquire a neighborhood point set of the reference point as a second neighborhood point set; Matching the pixel points in the first neighborhood point set and the second neighborhood point set, and obtaining the number of successfully matched pixel point pairs; If the number of successfully matched pixel pairs exceeds a preset second threshold, it is determined that the feature point and the reference point are correctly matched.

5. The method according to claim 1, characterized in that During the video playback, capturing an image of a preset area in the video screen includes: Acquire a preset reference image and its corresponding preset area; Feature point recognition is performed on the reference image to obtain a reference image feature set.

6. An image recognition device, characterized in that: include: A screenshot unit is used to capture an image of a preset area in the video screen during video playback; A feature unit, used to obtain a feature point set of the image and a descriptor of a feature point in the feature point set; A matching unit, configured to obtain, according to the descriptor of the feature point, a reference point having the shortest Hamming distance to the feature point in a preset reference image feature set; A detection unit, used to detect whether the feature point matches the reference point correctly; a determination unit, configured to determine that the image successfully matches the reference image if the number of correctly matched feature points in the feature point set exceeds a preset value; The feature unit is specifically used to perform feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image; The candidate feature point set is obtained by identifying the gray value information of the pixel points in the image and the corresponding reference points; the corner point response value of each feature point in the candidate feature point set is calculated respectively; according to the corner point response value, a preset number of feature points are screened out in the candidate feature point set to configure the feature point set of the image; The feature unit is further used to determine the corresponding reference image to be identified according to the position of the preset area, determine the required number of feature points according to the category of the identified reference image, and configure the required number to be the preset number.

7. The device according to claim 6, characterized in that The feature unit includes a candidate subunit and a screening subunit: The candidate subunit is used to perform feature point recognition on pixel points in the image to obtain a set of candidate feature points corresponding to the image; The screening subunit is used to screen the feature points in the candidate feature point set according to a preset rule to obtain the feature point set of the image.

8. The device according to claim 6, characterized in that The characteristic unit includes an identification subunit: The identification subunit is used to determine the reference point corresponding to the pixel point according to a preset radius with the pixel point in the image as the center of the circle; obtain the gray value information of the pixel point and its corresponding reference point, and determine whether the pixel point is a feature point according to the gray value information; if the pixel point is a feature point, configure the pixel point into the feature point set of the image.

9. The device according to claim 6, characterized in that The device also includes a hash unit: The hash unit is used to obtain a hash string of the image, and calculate a Hamming distance between the hash string and a preset reference image hash string; if the Hamming distance exceeds a preset first threshold, it is determined that the image fails to match the reference image; The feature unit is further configured to obtain a feature point set of the image and a descriptor of a feature point in the feature point set if the Hamming distance is less than a preset first threshold.

10. The device according to claim 6, characterized in that The detection unit is specifically used for: The neighborhood point set of the feature point is obtained as a first neighborhood point set, and the neighborhood point set of the reference point is obtained as a second neighborhood point set; pixel points in the first neighborhood point set and the second neighborhood point set are matched, and the number of successfully matched pixel point pairs is obtained; if the number of successfully matched pixel point pairs exceeds a preset second threshold, it is determined that the feature point is correctly matched with the reference point.

11. The device according to claim 6, characterized in that The device also includes a reference unit: The reference unit is used to obtain a preset reference image and a corresponding preset area; perform feature point recognition on the reference image to obtain a reference image feature set.

12. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the image recognition method according to any one of claims 1 to 5.

Citation Information

Cited By

  • Apparatus and method of video tracking

    EP4704049A1