Image frame position detection method and device, electronic equipment and storage medium

Abstract

The application discloses an image frame position detection method and device, electronic equipment and a storage medium. The method obtains a target image; a specific region template is used to locate position information of a specific region in the target image; and frame position information of a region to be positioned is determined according to a position proportional relationship between the specific region and the region to be positioned in the target image and the position information. The method only needs the specific region template in the whole process of detecting the image frame position, does not need a large amount of training data with accurate frame position annotation, and does not need to train a detection network model after obtaining the training data, so that the difficulty of extracting the image frame position is reduced, the image frame position detection cost is reduced, and since the program of the detection network is very large, the execution efficiency of the detection network is not high. In the application, only the specific region template is used to detect the frame position information of the region to be positioned, the process is simple, and the execution efficiency is high.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image border position detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of live video streaming and short videos, people's demands for video display content are becoming increasingly diversified. For example, multiple image regions often exist on the same display interface, and one of these regions needs to be displayed in full screen on its own. Alternatively, users may need to extract the desired region from an interface containing multiple image regions. In all these scenarios, it is necessary to determine the border position information of the desired region from the entire image, and then extract the region based on this border position information for subsequent use.

[0003] Existing technologies, when dealing with the aforementioned process of determining border position information, typically require using trained detection networks or other network models to detect the entire image displayed on the screen and calculate the border position information of the desired region. For example, a deep learning-based detection network can be used to detect the image and regress the border position of the desired region. However, this method requires training data with accurate border position annotations to train the detection network model; however, such data annotation is costly, and the detection network is inefficient on CPUs. Therefore, how to quickly and efficiently detect image border positions is a technical problem that those skilled in the art need to solve. Summary of the Invention

[0004] In view of this, this application provides an image border position detection method, apparatus, electronic device and storage medium, which can directly obtain the border position information of the region to be located in the target image by using only a specific region template, without any labeled data or training of detection networks or other models, and can achieve zero-shot fast image border position detection.

[0005] To achieve the above objectives, this application provides an image border position detection method, comprising:

[0006] Acquire the target image;

[0007] Using a specific region template, the location information of a specific region in the target image is located;

[0008] Based on the positional ratio between the specific region and the region to be located in the target image, and the positional information, the border position information of the region to be located is determined.

[0009] In one possible implementation, determining the border position information of the region to be located based on the positional ratio between the specific region and the region to be located in the target image, and the position information, includes:

[0010] Based on the positional ratio between the specific region and the region to be located in the target image, and the positional information, the initial border position information of the region to be located is obtained;

[0011] The initial border position information is processed using the mean algorithm to obtain the border position information of the region to be located.

[0012] In another possible implementation, extracting the target image from the received video includes:

[0013] Extract the video to be extracted from the received video; the video to be extracted is a video segment of a preset duration from the middle of the video.

[0014] Extract the target image from the video to be extracted.

[0015] In another possible implementation, the step of locating the positional information of a specific region in the target image using a specific region template includes:

[0016] Feature information of a predetermined region and a specific region template in the target image is extracted respectively;

[0017] Based on the specific region template and the feature information of the predetermined region, the location information corresponding to the specific region in the target image is determined.

[0018] In another possible implementation, determining the location information corresponding to the specific region in the target image based on the specific region template and the feature information of the predetermined region includes:

[0019] The nearest neighbor method is used to match the specific region template with the feature information of the predetermined region, and the number of matched feature point pairs is counted.

[0020] When the number exceeds a preset threshold, the location information corresponding to a specific region in the target image is obtained by using the matched feature point pairs.

[0021] In another possible implementation, obtaining the location information corresponding to a specific region in the target image using the matched feature point pairs includes:

[0022] Solve for the homography matrix of the matched feature point pairs;

[0023] The homography matrix is ​​used to perform a projection transformation on the location of a specific region in the target image to obtain the location information of the specific region in the target image.

[0024] In another aspect, this application provides an image border position detection device, comprising:

[0025] The acquisition module is used to acquire the target image;

[0026] A specific region positioning module is used to locate the position information of a specific region in the target image using a specific region template;

[0027] The image border positioning module is used to determine the border position information of the region to be positioned based on the positional ratio between the specific region and the region to be positioned in the target image, as well as the position information.

[0028] In another aspect, this application also provides an electronic device, comprising:

[0029] Memory, used to store computer programs;

[0030] A processor is configured to implement the image border position detection method of any embodiment of this application when executing the computer program.

[0031] In another aspect, this application also provides a storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the image border position detection method of any embodiment of this application.

[0032] As can be seen, in this embodiment, only a specific region template is needed to directly determine the position information of a specific region in the target image. After obtaining the position information of the specific region, the border position information of the region to be located in the target image can be directly calculated based on the positional ratio between the specific region and the region to be located, thus completing the image border position detection. That is, in the entire process of determining the image border position, this embodiment only requires a specific region template, without the need for a large amount of training data with accurate border position annotations, nor the need to train detection networks and other models after obtaining training data. Therefore, the difficulty of extracting image border position information can be reduced, and the high cost of annotated training data can be avoided. Furthermore, since the programs corresponding to detection networks are very large, the efficiency of running such detection networks is not high, making them unsuitable for large-scale video preprocessing. However, in this embodiment, only a specific region template is needed to realize the border position information detection of the region to be located. The process is simple, realizing a zero-sample border position information detection method for the region to be located. This method has high execution efficiency and is suitable for large-scale video preprocessing.

[0033] Accordingly, embodiments of this application also provide an image border position detection device, electronic device, and storage medium corresponding to the above-described image border position detection method, which have the above-described technical effects, and will not be described in detail here. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This diagram illustrates the hardware framework applicable to an image border position detection method according to an embodiment of this application.

[0036] Figure 2 A schematic diagram of the hardware framework applicable to another image border position detection method according to an embodiment of this application is shown;

[0037] Figure 3 This paper illustrates a flowchart of an image border position detection method according to an embodiment of the present application.

[0038] Figure 4 This illustration shows another flowchart of an image border position detection method according to an embodiment of this application;

[0039] Figure 5 This illustration shows a flowchart of a method for locating the position information of a specific region in a target image using a specific region template, according to an embodiment of this application.

[0040] Figure 6 This illustration shows a result of matching a small map region in a game area using a small map template in a target image frame of a game live stream video according to an embodiment of this application.

[0041] Figure 7 This illustration shows a schematic diagram of the location information of a small map area according to an embodiment of this application;

[0042] Figure 8 This illustration shows a schematic diagram of the result of the border position information of a game area according to an embodiment of this application;

[0043] Figure 9 This illustration shows a schematic diagram of a small map area template according to an embodiment of this application;

[0044] Figure 10 This illustration shows a flowchart of an embodiment of the present application for determining the border position information of the area to be located based on positional ratio and position information;

[0045] Figure 11 This illustration shows a flowchart of an embodiment of the present application for determining the border position information of the area to be located based on positional ratio and position information;

[0046] Figure 12 This illustration shows a schematic diagram corresponding to an application scenario embodiment of this application.

[0047] Figure 13 This diagram illustrates the structural composition of one embodiment of an image border position detection device according to an embodiment of this application. Detailed Implementation

[0048] Currently, when users face scenarios requiring automatic image cropping, they need to utilize trained detection networks and other models to detect the region to be located, regress the border position information of the region, and then complete the cropping of the desired area. For example, in MOBA game (such as Honor of Kings) live streams, the live video usually includes the game area, the streamer's avatar area, and advertising areas. In this application scenario, it is usually necessary to crop the game area from the live video. At this time, a detection network is needed to detect the live video and regress the border position information of the game area to complete the cropping. MOBA games (Multiplayer Online Battle Arena) are examples of such games. However, the training process of this detection network not only requires a large amount of training data with accurate border position annotations, but also has high training data acquisition costs. Furthermore, the processing efficiency of this detection network is relatively low, making it unsuitable for large-scale video preprocessing. The technical solution provided in this application embodiment can quickly and efficiently extract image border position information. It only needs to use a specific region template to determine the border position information of the region to be located in the target image, thus completing image border position detection.

[0049] To facilitate understanding, the hardware framework applicable to the image border position detection method of this application will be introduced first. (See also...) Figure 1 ,in, Figure 1 It shows a schematic diagram of the hardware framework applicable to the image border position detection method of this application.

[0050] Depend on Figure 1 It is known that the hardware framework may include: electronic device 10, wherein electronic device 10 may include: processor 11, memory 12, communication interface 13, input unit 14, display 15, and communication bus 16.

[0051] The processor 11, memory 12, communication interface 13, input unit 14, and display 15 all communicate with each other through the communication bus 16.

[0052] In this embodiment, the processor 11 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, an off-the-shelf programmable gate array, or other programmable logic devices. The processor can call programs stored in the memory 12. Specifically, the processor can perform the operations performed on the electronic device side in the following embodiments of the image border position detection method.

[0053] The memory 12 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory stores at least a program for implementing the following functions:

[0054] Acquire the target image;

[0055] Using a specific region template, the location information of a specific region in the target image can be located;

[0056] Based on the positional ratio between the specific region and the region to be located in the target image, as well as the positional information, the bounding box position information of the region to be located is determined.

[0057] In one possible implementation, the memory 12 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as image playback). The data storage area may store data created during computer use, such as user data, user access data, and audio / video data.

[0058] In addition, memory 12 may also include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0059] The communication interface 13 can be an interface for a communication module, such as the interface for a GSM module.

[0060] This application may also include a display 14 and an input unit 15, etc.

[0061] certainly, Figure 1 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than [other components]. Figure 1 More or fewer components as shown, or combinations of certain components.

[0062] in, Figure 1The electronic device 10 can be a terminal (such as a mobile terminal like a mobile phone or tablet computer, or a fixed terminal like a PC) or a server.

[0063] In this embodiment, the electronic device 10 can receive video or images sent by other external devices via the network through the communication interface 13; it can also obtain video or images through its own input unit 14 (such as a scanner) or memory.

[0064] Correspondingly, the processor 11 in the electronic device 10 can acquire the target image from the communication interface 13 or the input unit 14 through the communication bus 16, and call the program stored in the memory 12 to locate a specific region in the target image, obtain its position information, and combine the position information with the position ratio between the specific region and the region to be located to perform position information conversion to obtain the border position information of the region to be located in the target image. Then, based on the border position information, the image corresponding to the region to be located can be extracted from the target image, thereby realizing fast and efficient image border position detection.

[0065] In one possible scenario, to improve processing efficiency, in this embodiment, the electronic device 10 extracts target images from a video. For example, it extracts target images from the video to be extracted at equal intervals. The video to be extracted can be a video received by the electronic device 10, or a pre-set duration segment of a received video. Of course, the pre-set duration is not limited in this case.

[0066] In another possible scenario, to ensure the accuracy of the border position information of the region to be located in the target image, the aforementioned image border position detection method can be performed on multiple acquired target images to obtain multiple initial border position information of the region to be located. The final border position information of the region to be located can then be calculated based on these initial border position information. The electronic device 10 can also utilize an outlier removal algorithm to remove outlier initial border position information from the initial border position information, and then take the average of the removed initial border position information to obtain the border position information of the region to be located.

[0067] It is understood that the number of electronic devices is not limited in the embodiments of this application; multiple electronic devices can work together to complete the image border position detection method. In one possible scenario, please refer to... Figure 2 .Depend on Figure 2 It can be seen that the hardware framework may include: a first electronic device 101 and a second electronic device 102. The first electronic device 101 and the second electronic device 102 communicate with each other through a network 103.

[0068] In this embodiment, the hardware structure of the first electronic device 101 and the second electronic device 102 can be referred to Figure 1 The electronic device 10 in this embodiment can be understood as having two electronic devices 10, which interact with each other to realize the function of determining the position information of the image border. Furthermore, the form of the network 103 is not limited in this embodiment. For example, the network 103 can be a wireless network (such as WIFI, Bluetooth, etc.) or a wired network.

[0069] The first electronic device 101 and the second electronic device 102 can be the same type of electronic device, such as both being servers; or they can be different types of electronic devices, such as the first electronic device 101 being a terminal or a smart electronic device, and the second electronic device 102 being a server. In yet another possible scenario, please refer to [the relevant documentation / reference]. Figure 3 This application utilizes a powerful server as the second electronic device 102 to improve data processing efficiency and reliability, thereby enhancing the efficiency of image border position information detection. Simultaneously, it utilizes a low-cost, widely applicable terminal or smart electronic device as the first electronic device 101 to facilitate interaction between the second electronic device 102 and the user. The interaction process can be as follows: after acquiring the target image, the terminal sends the target image to the server. Upon receiving the target image, the server uses a specific region template to locate the position information of a specific region within the target image; based on the positional ratio between the specific region and the region to be located, and the position information, it determines the border position information of the region to be located in the target image. Of course, after calculating the border position information of the region to be located in the target image, the server can also use the located border position information as one of the tags of the target image (or a video of the target image captured), storing it together with the target image (or the corresponding video) in a database for subsequent retrieval.

[0070] Based on the above commonalities, see [link / reference] Figure 4 The flowchart illustrates an embodiment of an image border position detection method according to this application. The method of this embodiment may include:

[0071] S101. Obtain the target image.

[0072] This application does not limit the number of target images. Only one target image can be processed to obtain the border position information of the region to be located within the target image. Alternatively, a preset number of target images can be processed to ultimately obtain the border position information of the region to be located. Of course, this application does not limit the value of the preset number. This application also does not limit the method of determining the target image. In this application, one, multiple, or all images from the acquired original image or video can be directly used as the target image, such as extracting the target image from a received video. Alternatively, one, multiple, or all images from the acquired original image or video can be preprocessed and used as the target image. It is understood that this application does not limit the preprocessing process; it is related to the specific method selected in subsequent steps. That is, if subsequent steps can directly process the original image or video, then preprocessing of the original image or video is unnecessary. If the target image required by subsequent steps is a preprocessed image, then corresponding preprocessing of the extracted original image or video is required to obtain the target image. Of course, the electronic device can also directly receive the final target image obtained by other electronic devices in the above manner.

[0073] This application does not limit the method of extracting a portion of an image from the original image or video as the target image. The target image can be directly extracted from the original image or video at equal intervals. Alternatively, a portion can be directly cropped from the original image or video as the target image; for example, a preset duration segment of the video in the middle can be cropped as the target image, or the last preset duration segment of the video can be cropped as the target image. Of course, this application does not limit the value of the preset duration, and the "middle segment" mentioned in this application can refer to a preset duration segment of the video obtained by taking the preset frame after the beginning of the video as the start frame of the middle segment, and then continuing for a preset duration. Alternatively, the video to be extracted can be first cropped from the received video (this video to be extracted can be a preset duration segment of the video in the middle, or a preset duration segment of the video at the end, etc.); and then the target image can be extracted from the video to be extracted. Of course, the extraction can be performed at equal intervals or randomly; this embodiment does not limit this.

[0074] It should be noted that the method for extracting the target image in this embodiment can be selected according to the specific application scenario. For example, when the position of the border of the region to be located is consistent in every frame of the video, in order to improve processing efficiency while ensuring the accuracy of the border position information of the region to be located, only a certain number of video frames can be selected as target images for border position information detection of the region to be located. When not every frame of the received video contains the region to be located, in order to improve the efficiency of border position information detection of the region to be located, a video segment with a relatively concentrated region to be located can be extracted from the video. Of course, provided that the extraction rules are not contradictory, the corresponding extraction rules can be superimposed according to the actual application scenario to obtain the target image. For example, when the position of the border of the region to be located is consistent in every frame of the video, and not every frame of the video contains the region to be located, a video segment with a relatively concentrated region to be located can be extracted first, and then the video segment can be extracted at equal intervals to obtain the target image.

[0075] It is understood that the embodiments of this application do not limit the method of acquiring the original image or video. For example, it can be to directly receive the original image or video uploaded by other electronic devices through a network (which can be a wired network or a wireless network), or it can be to directly acquire the input original image or video from the local device, or it can be to acquire the original image or video through a connected USB flash drive.

[0076] S102. Using a specific region template, locate the position information of a specific region in the target image.

[0077] This application does not limit the selection method of the specific region, as long as the selected specific region has fixed characteristics and a certain positional ratio with the region to be located. The specific region having fixed characteristics ensures that it can be identified from the target image, thereby allowing the acquisition of its positional information. The certain positional ratio between the specific region and the region to be located allows the calculation of the border position information of the region to be located after obtaining the positional information of the specific region. Of course, this application does not limit the specific location of the specific region in the target image; for example, the specific region can be a part of the region to be located, or it can be any region in the target image other than the region to be located.

[0078] In this embodiment, the specific region template corresponds to the specific region. This can be understood as follows: when the selected specific region is a region that will not change (e.g., an icon in a target image whose features will not change), the specific region template is the specific region itself; for example, a specific region image can be cropped from an image as the specific region template. When the selected specific region has fixed features but also exhibits changes (e.g., a minimap exists in a game video; the road and other features on the minimap are fixed, but the hero icons appearing on the minimap in each game can change), the specific region template can be a specific region; for example, a specific region image can be cropped from any image as the specific region template. Alternatively, the specific region image with the clearest fixed features can be found from multiple images as the specific region template.

[0079] It should be noted that the number of specific regions is not limited in the embodiments of this application. There can be one or more specific regions. When there are multiple specific regions, if the specific regions are different, then a corresponding specific region template needs to be set for each specific region. If there are specific regions with the same image content but different positions in the target image, a corresponding specific region template can be set for these specific regions. Of course, a corresponding specific region template can also be set for each specific region.

[0080] Since the specific region template provided in this application embodiment has the same features or the same fixed features as the specific region, this application embodiment can match the specific region from the target image using the specific region template, and thus obtain the location information of the specific region. It should be noted that this application embodiment does not limit the specific method of matching the specific region from the target image using the specific region template. For example, the matching of the specific region template and the specific region can be achieved using a local feature matching method.

[0081] It should be noted that the embodiments of this application do not limit the size of the region in the target image that is matched with the specific region template. The entire target image can be directly matched with the specific region template; alternatively, a predetermined region within the target image can be matched with the specific region template. Of course, the size of the predetermined region is not limited in the embodiments of this application, but it must include the specific region in the target image. This can be understood as the size of the predetermined region being larger than the size of the specific region, but smaller than the overall size of the target image. Naturally, if the predetermined region accurately includes the specific region, the smaller the predetermined region, the fewer feature points need to be matched, and correspondingly, the speed of locating the position information of the specific region will increase. Therefore, to improve the efficiency of determining the image border position information, only the predetermined region in the target image can be matched with the specific region template. That is, the embodiments of this application can extract the feature information of the predetermined region and the specific region template in the target image respectively; based on the feature information of the specific region template and the predetermined region, the position information corresponding to the specific region in the target image can be determined.

[0082] This application does not limit the method for extracting feature information of a predetermined region and a specific region template in the target image. For example, SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Features), ORB (Oriented Fast and Rotated BRIEF), and BRISK (Binary Robust Invariant Scalable Keypoints) algorithms can be used to extract feature information of the predetermined region and the specific region template. Users can choose a suitable algorithm based on the image conditions of the specific region in the actual application scenario, the user's detection efficiency of the image border position information, and the hardware processing capabilities of the electronic device. Among them, SURF is an accelerated version of the SIFT algorithm; ORB and BRISK algorithms are much less robust than SIFT, but can greatly improve speed.

[0083] It is understandable that after extracting the feature information of the predetermined region and the specific region template, a feature matching operation needs to be performed to complete the matching of the specific region in the target image. This application embodiment does not limit the method of matching the feature information of the specific region template and the predetermined region. For example, the nearest neighbor method can be selected to match the feature information of the specific region template and the predetermined region. Of course, the user can choose the corresponding feature matching method according to the selected feature extraction method.

[0084] It should be noted that when there is only one target image in this embodiment, two situations may occur during feature matching. One situation is that a specific region is matched from a predetermined region in the target image using a specific region template. Of course, when a specific region is successfully matched, this embodiment can use the matched feature point pairs to calculate the position information corresponding to the specific region in the target image. The other situation is that the specific region template fails to match the specific region from the predetermined region in the target image. This embodiment does not limit the subsequent operations in this situation. For example, it could directly output a prompt indicating that the image border position information detection failed; it could output a prompt to reacquire the target image; it could combine the above operations; or it could simply end the current operation. Users can choose according to their actual situation.

[0085] When there are multiple target images in this embodiment, two situations may occur during feature matching. One situation is that a specific region is matched from a predetermined region in the target images using a specific region template; that is, a specific region is matched in all or some of the target images. Of course, when a specific region is successfully matched, this embodiment can use the matched feature point pairs to calculate the position information corresponding to the specific region in each successfully matched target image. The other situation is that no specific region can be matched from the predetermined region in the target images using the specific region template. This embodiment does not limit the subsequent operations in this situation. For example, it can directly output a prompt message indicating that the image border position information detection failed; it can output a prompt message to reacquire the target image; it can combine the above operations; or it can directly end the current operation. The user can choose according to the actual situation.

[0086] It is understood that in this embodiment, the location information of the specific region in the target image is calculated only when a specific region is successfully matched within a predetermined region. This embodiment does not limit the method for determining whether a specific region has been successfully matched within a predetermined region; it is adapted to the selected feature matching method. For example, when using the nearest neighbor method to match the specific region template with the feature information of the predetermined region, the number of successfully matched feature point pairs can be used to determine whether a specific region has been successfully matched within the predetermined region. That is, if the number of successfully matched feature point pairs is greater than a preset threshold, then the specific region is considered to exist within the predetermined region of the target image; if the number of successfully matched feature point pairs is not greater than the preset threshold, then the specific region is considered not to exist within the predetermined region of the target image. Of course, this embodiment does not limit the value of the preset threshold; users can determine and modify this value according to actual circumstances.

[0087] It should be noted that the embodiments of this application are not limited to using successfully matched feature point pairs to calculate the location information corresponding to a specific region in the target image. For example, the corresponding homography matrix can be calculated using the successfully matched feature point pairs, and the location information of the specific region in the target image can be determined based on the homography matrix (the location information can be the coordinate information of the specific region in the target image). Here, the homography matrix of a plane is defined as a projection mapping from one plane to another.

[0088] S103. Determine the bounding box position information of the region to be located based on the positional ratio between the specific region and the region to be located in the target image, as well as the positional information.

[0089] After determining the position information of a specific region in the target image, this embodiment of the application can convert the position information according to the positional ratio between the specific region and the region to be located in the target image to obtain the border position information of the region to be located in the target image. It should be noted that the positional ratio in this embodiment can refer to the positional ratio between the specific region and the region to be located in the target image; or it can refer to the positional ratio between the region to be located and the specific region. Of course, this embodiment of the application does not limit the positional relationship between the specific region and the region to be located in the target image. For example, the specific region can be a part of the region to be located; the specific region can also be a part outside the region to be located. As long as there is a certain positional ratio between the two, the border position information corresponding to the region to be located can be calculated according to the positional ratio of the specific region. It should also be noted that this embodiment of the application does not limit the size of the region to be located in the target image; that is, the region to be located can be a small part of the target image or the entire target image.

[0090] It is understandable that when only the location information of a specific region is obtained, the border position information of the region to be located, which is directly calculated based on the location information and the position ratio, can be used directly as the final border position information of the region to be located. When the location information of multiple specific regions is obtained, the border position information of the region to be located corresponding to each specific region can be calculated based on each location information and the position ratio, and then the final border position information of the region to be located can be obtained based on the border position information of the region to be located corresponding to each specific region. Furthermore, in order to improve the accuracy of the final border position information of the region to be located, this application embodiment can obtain the initial border position information of the region to be located based on the position ratio relationship between the specific region and the region to be located in the target image, as well as the location information; and process the initial border position information using an averaging algorithm to obtain the border position information of the region to be located. That is, this application embodiment reduces the error of the final border position information by averaging the obtained initial border position information, thereby improving the accuracy of the final border position information of the region to be located.

[0091] Of course, the embodiments of this application do not limit the method of processing the initial border position information using the mean algorithm to obtain the border position information of the region to be located; that is, the embodiments of this application do not limit the mean algorithm. For example, the initial border position information of the region to be located corresponding to each specific region can be directly averaged using the mean algorithm to obtain the final border position information of the region to be located. Correspondingly, the mean algorithm can be an average calculation method. Alternatively, the mean algorithm can be used to filter out the initial border position information with smaller errors from the initial border position information of the region to be located corresponding to each specific region, and then the average of the filtered initial border position information can be taken to obtain the final border position information of the region to be located. Correspondingly, the mean algorithm can be a linear regression algorithm (using regression analysis methods in mathematical statistics to fit data), or an outlier removal algorithm (a distance-based outlier removal algorithm that can remove bad data). Alternatively, the initial border position information of the region to be located corresponding to each specific region can be sorted in ascending order using the mean algorithm, and the average of the selected middle preset number of initial border position information can be taken as the final border position information of the region to be located. Of course, the embodiments of this application do not limit the value of the preset quantity.

[0092] As can be seen, the embodiments of this application can automatically extract the border position information of the region to be located in the target image. This method enables automatic annotation of the border position information of the region to be located in the target image, for example, adding a border position information label to the video. Furthermore, based on this annotation, the region to be located can be quickly extracted from subsequently uploaded videos of this type, removing irrelevant content to improve the accuracy and reliability of subsequent detection and recognition of specific elements in the region. For example, when the region to be located is a game area, it can improve the accuracy of subsequent recognition of key elements such as heroes and event announcements within the game area. In other words, the embodiments of this application provide an efficient preprocessing method for image key element detection and recognition. Of course, the specific label format is not limited in the embodiments of this application.

[0093] In this embodiment, only a specific region template corresponding to a specific region is needed to locate the position information of a specific region in the target image. Then, based on the positional ratio between the specific region and the region to be located in the target image, the border position information of the region to be located in the target image can be directly calculated, completing the image border position detection process. It can be seen that this embodiment only requires a specific region template to directly determine the position information of a specific region in the target image. It does not require obtaining a large amount of training data with precise border position annotations before detecting the image border position, nor does it require training the detection network after obtaining the training data. Therefore, this embodiment can reduce the difficulty and cost of extracting image border positions. Furthermore, since detection network programs are very large and their execution efficiency is low, they are not suitable for large-scale video preprocessing. This embodiment only requires a specific region template to locate the position information of a specific region, simplifying the process and realizing a zero-sample method for determining the border position information of the region to be located—that is, an efficient and robust method for determining border position information. This method has high execution efficiency and is suitable for large-scale video preprocessing. Furthermore, in this embodiment, only the positional ratio between the specific region and the region to be located in the target image is needed to directly calculate the border position information of the region to be located in the target image. Therefore, when the specific region is smaller than the region to be located, the efficiency of image border position information detection can be further improved. That is, in this embodiment, the efficiency of image border position detection can also be improved by locating a large region with a small region.

[0094] In one possible scenario, this application provides a process for locating the positional information of a specific region in a target image using a specific region template; please refer to [link / reference needed] for details. Figure 5 The specific execution process is as follows:

[0095] S501. Extract the feature information of the predetermined region and the specific region template in the target image respectively.

[0096] In this embodiment, to improve the detection efficiency of location information of a specific region in the target image, only feature information of a predetermined region containing the specific region is extracted from the target image. The selection of the predetermined region, the specific region, and the specific region template can be made by the user according to the actual situation, and the relevant selection criteria can be referred to the descriptions in the above embodiments. For example, when the target image is a game live stream image, and the region to be located in this application that needs to determine the border position information is the game area in the game live stream image, then the specific region and the specific region template can be the small map area in the game area. Generally, the small map in the game is located in the upper left corner of the game image, and the upper left quarter of the game live stream image can certainly contain this small map area. Therefore, the predetermined region can be selected from the upper left quarter of the game live stream image.

[0097] It should be noted that the embodiments of this application do not limit the method of extracting feature information of the predetermined region and feature information of the specific region template. For example, the SIFT algorithm can be used to extract feature information of the predetermined region and the specific region template; the SURF algorithm can be used to extract feature information of the predetermined region and the specific region template; the ORB algorithm can be used to extract feature information of the predetermined region and the specific region template; the BRISK algorithm can be used to extract feature information of the predetermined region and the specific region template, etc. Of course, the embodiments of this application do not limit the specific feature information extraction process, and users can perform corresponding feature extraction operations according to the selected feature extraction algorithm.

[0098] For example, the process of extracting feature information from a predetermined region and a specific region template in a target image using the SIFT algorithm can be as follows: Corner features are detected in both the specific region template and the predetermined region, and the scale, position, and orientation information of the detected corner points are used to represent the feature vectors (i.e., feature information, which can also be represented as SIFT feature vectors) of the corner points. For example, using... This represents the feature vector of the k-th corner point of a specific region template. This represents the feature vector of the k-th corner point in the predetermined region. The corner point is the feature point. The SIFT algorithm is a general algorithm for detecting local features. SIFT features are scale-invariant, meaning they can still achieve good detection results even when the rotation angle, image brightness, or shooting angle is changed.

[0099] S502. Use the nearest neighbor method to match the template of a specific region with the feature information of a predetermined region, and count the number of matched feature point pairs.

[0100] This application embodiment utilizes the nearest neighbor method to match a specific region template with the feature information of a predetermined region. The specific matching process requires performing corresponding matching operations based on the content of the obtained feature information. For example, when using the nearest neighbor method to match the SIFT feature vectors of M feature points of a specific region template... SIFT feature vectors of N feature points in a predetermined region The process of matching and counting the number of matched feature points can be as follows: Calculate the Euclidean distance *d* between the SIFT feature vector of each feature point in the specific region template and the SIFT feature vector of any feature point in the predetermined region; for any feature point in the specific region template, find the two feature points in the predetermined region with the closest Euclidean distance *d*. If the distance between the closest and second-closest feature points is less than a set threshold, then the feature point in the specific region template and the feature point in the predetermined region with the closest Euclidean distance *d* are considered a successfully matched feature point pair. Here, the Euclidean distance *d* serves as a similarity metric between the feature points in the specific region template and the feature points in the predetermined region. For example, the feature points in the specific region template... Feature points of the predetermined area Similarity determination metric This application does not limit the value of the ratio threshold (e.g., the ratio threshold can be set to an empirical value of 0.7). The smaller the ratio threshold, the fewer the number of successfully matched feature point pairs will be, and correspondingly, the matched feature point pairs will be more stable and reliable. Users can set the ratio threshold according to the actual application scenario and the required reliability of image border position detection. Of course, users can also adjust the ratio threshold according to the actual border position detection of the area to be located. For example, please refer to the appendix. Figure 6 This paper presents a schematic diagram illustrating the result of matching a minimap region within the game area to a target image frame from a game livestream video using a minimap region template. In this diagram, a single frame of the livestream game image is used as the target image, the minimap region as the specific region, the minimap region template as the specific region template, and the game area as the region to be located. The left side of the diagram represents the minimap region template. Straight lines connecting the minimap region template to the frame indicate successful matching of feature points at corresponding positions in the minimap region template with their corresponding positions in the frame. Using the positions of these successfully matched feature points, the specific location of the minimap region within the frame can be obtained.

[0101] S503. Determine whether the quantity is greater than a preset threshold; if the quantity is not greater than the preset threshold, proceed to step S504; if the quantity is greater than the preset threshold, proceed to step S505.

[0102] S504, Output not matched to a specific region.

[0103] S505. Solve for the homography matrix of the matched feature point pairs.

[0104] It should be noted that the specific value of the preset threshold is not limited in this embodiment (e.g., the preset threshold can be set to 15). A larger preset threshold value results in higher reliability of the detected specific region, reducing the false detection rate but increasing the false negative rate. Conversely, a smaller preset threshold value reduces the reliability of the detected specific region, reducing the false negative rate but increasing the false positive rate. Therefore, users can set the preset threshold according to the actual application scenario and the required reliability of image border position detection. Alternatively, users can adjust the preset threshold based on the actual border position detection of the region to be located. When the number of detected borders exceeds the preset threshold, the specific region is considered to exist within the predetermined region of the target image; when the number of detected borders is less than the preset threshold, the specific region is considered not to exist within the predetermined region of the target image. For example, when the preset threshold is 15 and the specific region is a small map region, if the number of matched feature point pairs is higher than 15, it is determined that a small map exists within the predetermined region of the target image; if the number of matched feature point pairs is less than 15, it is determined that a small map does not exist within the predetermined region of the target image.

[0105] Of course, if a specific region is not found within the predetermined area of ​​the target image, a prompt message indicating that the specific region was not matched can be output according to step S504. The output method of this prompt message is not limited; it can be output via a display screen or via voice, etc. Alternatively, the detection of the current target image can be terminated directly, and the detection process for the next frame of the target image can begin.

[0106] S506. Use the homography matrix to perform a projection transformation on the location of a specific region in the target image to obtain the location information of the specific region in the target image.

[0107] In this embodiment, when a specific region is detected in a predetermined area of ​​the target image, the position information of that specific region in the target image can be calculated. That is, the position information of the specific region in the target image is calculated based on the matched feature point pair information. To improve the reliability and efficiency of detecting the position information of a specific region in the target image, this embodiment utilizes a homography matrix to solve for the position information of the specific region in the target image. This process can be as follows: First, solve for the homography matrix H of the matched feature point pair information. Then, use the homography matrix H to perform a projection transformation on the position src(x,y) of the specific region in the target image to obtain the position information dst(x,y) of the specific region in the target image, where dst(x,y) = src(x,y) * H.

[0108] It should be noted that the content of the location information in this embodiment is not limited. It can be the coordinate information of each point in a specific area; it can also be the coordinate information of the border points in a specific area; when the specific area is a regular shape such as a matrix, it can also be the coordinate information of all or a specified portion of the inflection points of the border in the specific area, etc. The content of the location information of the specific area in this embodiment is related to the shape and positional ratio of the area to be detected. Users can determine the content of the location information of the specific area required based on actual conditions.

[0109] For example, please refer to the appendix. Figure 7 Appendix Figure 8 and appendix Figure 9 The diagram presents schematic diagrams of the location information of the minimap area, the border location information of the game area, and a template of the minimap area. A single frame of the live game image is used as the target image, the minimap area (a small rectangle) as the specific region, and the game area (a large rectangle) as the region to be located. Since the minimap area is located at the top left corner of the game area, and their top-left vertices coincide, and both the minimap and large map areas are rectangles with a proportional relationship, knowing only the coordinates of the top-left and bottom-right vertices of the minimap area, along with their proportional relationship, is sufficient to determine the coordinates of the top-left and bottom-right vertices of the game area. Furthermore, because the game area is a matrix, these two diagonal coordinates uniquely determine the screen position of the game area, thus obtaining the border location information of the game area. Figure 7 The coordinates of the top-left and bottom-right vertices of the minimap area are given, where the starting position of the coordinate system is the top-left corner of the target image, i.e., the game live stream image. Figure 8 The coordinates of the top-left and bottom-right vertices in the game area are given. Of course, the starting position of the coordinate system is still the top-left corner of the target image, i.e., the game live stream image. Figure 9 A schematic diagram of a small map area template is provided.

[0110] As can be seen, in this embodiment, it is not necessary to acquire a large amount of training data with precise border position annotations before detecting the image border position, nor is it necessary to train the detection network after acquiring the training data. Only feature extraction algorithms are used to extract feature information from a predetermined region in the target image and a given specific region template. Then, the nearest neighbor method is used to determine the successfully matched feature point pairs when a specific region is successfully matched. Finally, the position information of the specific region in the target image is directly projected and transformed based on the homography matrix corresponding to the successfully matched feature point pairs. The calculation process is simple and efficient.

[0111] In another possible scenario, embodiments of this application provide a process for determining the bounding box position information of the region to be located based on the positional ratio between a specific region and the region to be located in the target image, as well as positional information; please refer to [link / reference needed] for details. Figure 10 The specific execution process is as follows:

[0112] S1001. Based on the positional ratio between the specific region and the region to be located in the target image, as well as the positional information, obtain the initial bounding box position information of the region to be located.

[0113] S1002. Use the outlier removal algorithm to remove outliers from the initial border position information, and take the average of the removed initial border position information to obtain the border position information of the region to be located.

[0114] In this embodiment, when a specific region is detected in multiple target images, the location information of the specific region corresponding to each target image in which the specific region was detected is calculated. Then, based on the positional ratio between the specific region and the region to be located, the location information is converted to obtain the initial border location information of the region to be located for each target image in which the specific region was detected. That is, in this embodiment, a corresponding initial border location information is calculated for each target image in which a specific region is detected. It should be noted that this embodiment does not limit the initial border location information or its content, as long as it can uniquely determine the position of the region to be located in the target image. For example, when the region to be located is rectangular, determining the coordinates of the two diagonals is sufficient to uniquely determine the position of the region to be located in the image.

[0115] For example, when the target image is a live stream image of the game Honor of Kings, and the area to be located is the game area within the live stream image, the specific area and its template can be the small map area within the game area. When the predetermined area can be selected as the upper left quarter of the live stream image, the positional ratio between the small map area and the game area is as follows: the upper left corner of the small map area and the upper left corner of the game area have the same coordinates. The ratio of the screen width of the small map area to the screen width of the game area in the Honor of Kings live stream image is fixed at 0.3167, and the aspect ratio of the game area is 16:9. Correspondingly, the process of calculating the initial border position information can be: the position information of the small map area corresponding to the k-th live stream image is: the coordinates of the upper left vertex of the small map area. and the coordinates of the bottom right corner vertex Furthermore, since the coordinates of the top-left corner of the minimap area are the same as the coordinates of the top-left corner of the game area, the coordinates of the top-left corner of the minimap area are identical to those of the top-left corner of the game area. Therefore, the coordinates of the top-left corner of the game area are: In the Honor of Kings game live stream, the ratio of the screen width of the minimap area to the screen width of the game area is fixed at 0.3167, which is combined with the width value of the minimap area. according to This gives you the width of the game area, which is also the x-coordinate of the bottom right corner of the game area. Furthermore, since the aspect ratio of the game area is 16:9, according to This allows us to obtain the ordinate of the bottom right corner vertex of the game area. Then, based on the coordinates of the top left corner vertex and the bottom right corner vertex of the game area, the initial border position information of the game area in the target image of that frame can be determined.

[0116] In this embodiment, to avoid the influence of falsely detected location information of specific regions on the subsequent border location information of the region to be located, the accuracy of the border location information of the region to be located is improved. This embodiment utilizes an outlier removal algorithm to remove outliers from the initial border location information, and then averages the remaining initial border location information to obtain the border location information of the region to be located. The following example illustrates the processing of the outlier removal algorithm: when the target image is a live stream image of the game Honor of Kings, the region to be located is the game area in the live stream image, and the specific region and its template can be the small map area within the game area, the above steps are used to calculate the initial border location information of the game area corresponding to the video frame (i.e., the target image) that successfully matches the small map area (e.g., the target image). Record the coordinates of the top-left vertex of the game area in the k-th target image. Record the coordinates of the bottom right vertex of the game area in the k-th target image, and you can obtain the set of coordinates for all initial bounding box positions. Calculate the x-coordinates of the top left vertex. The coordinate values ​​are sorted by frequency from highest to lowest to obtain a set of candidate coordinate values, such as the set... The coordinate values ​​in the middle are determined by Three candidate values ​​form a candidate coordinate value set. Coordinate values ​​in the candidate coordinate value set are eliminated sequentially from highest to lowest frequency. For each eliminated candidate value, the variance of the remaining candidate values ​​is calculated. If the variance exceeds a pre-set threshold, the elimination continues until the variance of the remaining candidate values ​​falls below the pre-set threshold. At this point, elimination stops, and the mean of all remaining candidate values ​​is calculated as the detection value for that coordinate point. That is, when the calculated variance is less than the pre-set threshold, it is considered that the outlier in the x-coordinate set of the top-left vertex has been eliminated, and the mean of the remaining coordinate values ​​in the x-coordinate set of the top-left vertex is calculated as the x-axis coordinate of the top-left vertex in the final game area border position information. Following the above process, the y-coordinate set of the top-left vertex is then... The set of x-coordinates of the bottom right vertex The set of y-coordinates of the bottom right vertex By performing the above process, we obtain the vertical coordinate of the top-left vertex, the horizontal coordinate of the bottom-right vertex, and the vertical coordinate of the bottom-right vertex in the final game area border position information.

[0117] Of course, this application does not limit the value of the set variance threshold; for example, the set variance threshold can be set to 5. Furthermore, since the final mean coordinate value may be a decimal, the user can set the length of the final mean coordinate value. For example, the coordinates in the border position information of the area to be located can be obtained by rounding down each of the obtained mean coordinate values, or by rounding each of the obtained mean coordinate values ​​to two decimal places. This application does not limit this approach.

[0118] In this embodiment, when multiple initial border position information of regions to be located are detected, in order to improve the accuracy of the border position information, this embodiment performs statistical intervention on the calculated initial border position information to eliminate false detections, thereby improving the accuracy of the border position information of regions to be located. This embodiment only performs border detection on a certain number of video frames and performs statistical intervention on the detection results to eliminate false detections, thereby locating the border position information of the regions to be located. This approach can greatly improve the processing efficiency of border detection in the video and save computing and storage resources.

[0119] In another possible scenario, embodiments of this application provide a process for acquiring a target image; please refer to [reference needed]. Figure 11 The specific execution process is as follows:

[0120] S1101. Extract the video to be extracted from the received video; the video to be extracted is a video segment of a preset duration in the middle of the video.

[0121] S1102. Extract the target image from the video to be extracted.

[0122] It is understandable that in user-uploaded videos such as live streams, the video header typically corresponds to the start phase, and the video footer typically corresponds to the end phase; both of these have significant uncertainties. For example, the video header might contain advertisements, intros, or the preparation phase for starting a game, while the video footer might contain summaries, outros, or the exit phase after a game ends. Therefore, both the video header and footer may contain images that do not contain the region to be located. To avoid selecting images that do not contain the region to be located as target images and to improve the efficiency of image border position detection, this embodiment of the application will extract the video to be extracted from the received video; the video to be extracted is a video of a preset duration in the middle section of the video. That is, in this embodiment of the application, the video to be extracted is the video obtained after removing the header and footer of the received video. Of course, this embodiment of the application does not limit the length of the video header and footer; users can determine the length of the extracted middle section of the video according to the specific application scenario.

[0123] It should be noted that the embodiments of this application are not limited to the method of extracting the video to be extracted from the received video. The video to be extracted can be obtained by removing the video whose beginning and end durations each account for a first preset value, and the video whose ending duration accounts for a second preset value. For example, the video to be extracted can be obtained by removing the beginning and end durations (each accounting for 20%) from the received video, and the remaining 60% of the video in the middle. Alternatively, the video to be extracted can be obtained by taking the first preset frame or a preset duration image from the received video as the starting frame of the middle segment, and continuing for a preset duration. For example, the video to be extracted can be obtained by taking 1 minute from the beginning of the received video as the start of the preset duration and continuing for 5 minutes. Alternatively, the video to be extracted can be obtained by directly extending a preset frame or preset duration to the left and right from the midpoint of the received video. For example, if a video with 101 frames is received, starting from frame 51, counting 20 frames to the left and then 20 frames to the right, the resulting 41 frames can be used as the video to be extracted. Of course, the embodiments of this application do not limit the specific values ​​of the first preset value, the second preset value, the preset frame, and the preset duration.

[0124] Generally, if the border position information of the same region to be located in the same video is consistent, it is not necessary to perform border position detection on all frames of the acquired video. Only a certain number of video frames (i.e., the target image) need to be detected. Therefore, in order to improve the efficiency of border position detection processing of the region to be located in the video and save computing and storage resources, this application embodiment needs to extract target images from the video after determining the video to be extracted. This application embodiment does not limit the specific method of extracting target images from the video to be extracted. It can be to directly extract target images from the video to be extracted at equal intervals; or it can be to first set the number of target images to be detected, and then uniformly or randomly extract the corresponding number of target images from the video to be extracted. Of course, this application embodiment does not limit the specific values ​​of equal intervals and the number of target images to be detected.

[0125] In this embodiment, by extracting a portion of the received video as the video to be extracted, and then extracting the target image from the video to be extracted, the processing of images that do not contain the region to be located can be minimized. Furthermore, since only a portion of the video frames are processed, the number of target images to be detected can be further reduced while ensuring the reliability of the detection of the border position information of the region to be located. Therefore, the method for obtaining target images provided in this embodiment can significantly improve the processing efficiency of video border detection and save computing and storage resources of electronic devices.

[0126] For easier understanding, please refer to Figure 12 This paper will introduce one application scenario of this solution. The following describes the process of determining the game area border position information in the video frame of the game live broadcast, using the terminal and server, and MOBA game (Multiplayer Online Battle Arena) such as Honor of Kings as the processing object.

[0127] In the game livestream videos of Honor of Kings, a small map area typically exists within the game area. This small map area has a similar shape and relatively fixed characteristics, and its position is generally in the upper left corner of the game area. The size of the small map area is also proportional to the size of the game area. Therefore, after locating the position information of the small map area, the coordinates of the upper left corner vertex of the small map area can be directly used as the coordinates of the upper left corner vertex of the game area. Then, using the positional ratio between the small map area and the game area, the coordinates of the lower right corner vertex of the game area can be determined, thus obtaining the position information of the game area's border. In this application scenario embodiment, the game livestream image can be used as the target image, the small map area as the specific area, the small map area template as the specific area template, the game area as the area to be located, and the upper left quarter of the target image as the predetermined area. The specific processing procedure is as follows:

[0128] A user uploads a live stream video of the game Honor of Kings to the server via a terminal. The server first extracts a preset number of video frames from the received live stream video as target images. Then, it uses the SIFT algorithm to extract feature information from the upper left quarter region of the target image and the minimap region template. The nearest neighbor method is used to match the feature information of the minimap region template with the feature information of the upper left quarter region of the target image, and the number of matched feature point pairs is counted. When the number is greater than 15, the homography matrix of the matched feature point pairs is calculated. The homography matrix is ​​then used to project the position of the minimap region in the target image to obtain the position information of the minimap region in the target image. Based on the positional ratio between the minimap region and the game area, and the position information of the minimap region, the initial border position information of each target image with detected minimap regions is calculated. An outlier removal algorithm is used to remove outliers from the initial border position information, and the average of the removed initial border position information is taken to obtain the final border position information of the game area. The border position information of the located game area can also be used as one of the tags of the Honor of Kings live stream video and stored in the database along with the Honor of Kings live stream video content.

[0129] Among them, *Honor of Kings* is a MOBA mobile game developed and operated by Tencent Games' TiMi Studio Group, running on Android, iOS, and Nintendo Switch platforms. The game's gameplay primarily focuses on competitive battles, with players engaging in 1v1, 3v3, and 5v5 PVP battles. Players can also participate in the game's adventure mode, a PVE challenge mode, and, after meeting certain conditions, participate in ranked matches. Game live streaming refers to broadcasting gameplay simultaneously using internet technology.

[0130] On the other hand, this application also provides an image border position detection device. For example, see... Figure 13 This diagram illustrates the structural composition of an embodiment of an image border position detection device according to this application. The device of this embodiment can be applied to electronic devices as described in the above embodiments. The device includes:

[0131] Acquisition module 101 is used to acquire the target image;

[0132] The specific region positioning module 102 is used to locate the position information of a specific region in the target image using a specific region template.

[0133] The image border positioning module 103 is used to determine the border position information of the region to be positioned based on the positional ratio between a specific region and the region to be positioned in the target image, as well as the positional information.

[0134] Optionally, the acquisition module 101 may include:

[0135] The extraction submodule is used to extract the target image from the received video.

[0136] Optionally, the extracted submodules may include:

[0137] The extraction unit is used to extract the video to be extracted from the received video; the video to be extracted is a video segment of a preset duration in the middle of the video.

[0138] The extraction unit is used to extract the target image from the video to be extracted.

[0139] Optionally, the extraction unit may include:

[0140] Extraction subunits are used to extract target images from the video to be extracted at equal intervals.

[0141] Optionally, the specific area positioning module 102 may include:

[0142] The feature extraction submodule is used to extract feature information from a predetermined region and a specific region template in the target image, respectively.

[0143] The specific region location acquisition submodule is used to determine the location information corresponding to a specific region in the target image based on the specific region template and the feature information of the predetermined region.

[0144] Optionally, the feature extraction submodule may include:

[0145] The feature extraction unit is used to extract feature information of a predetermined region and a specific region template in the target image using the SIFT algorithm.

[0146] Optionally, the specific area location acquisition submodule may include:

[0147] The feature matching unit is used to match a specific region template with the feature information of a predetermined region using the nearest neighbor method, and to count the number of matched feature point pairs.

[0148] The specific region location acquisition unit is used to acquire the location information of a specific region in the target image by using the matched feature point pair information when the number exceeds a preset threshold.

[0149] Optionally, the specific area location acquisition unit may include:

[0150] The homography matrix sub-unit is used to solve for the homography matrix of the matched feature point pairs;

[0151] The specific region location acquisition sub-unit is used to perform a projection transformation on the specific region location in the target image using the homography matrix to obtain the location information of the specific region in the target image.

[0152] Optionally, the image border positioning module 103 may include:

[0153] The initial bounding box position information acquisition unit is used to obtain the initial bounding box position information of the region to be located based on the positional ratio between the specific region and the region to be located in the target image, as well as the positional information.

[0154] The image border localization unit is used to process the initial border position information using the mean algorithm to obtain the border position information of the area to be located.

[0155] On the other hand, this application also provides an electronic device that may include a processor and a memory. The relationship between the processor and the memory in this electronic device can be referenced... Figure 1 .

[0156] The processor of an electronic device is used to execute programs stored in the memory;

[0157] The memory of an electronic device is used to store programs, which are used at least for:

[0158] Acquire the target image;

[0159] Using a specific region template, the location information of a specific region in the target image can be located;

[0160] Based on the positional ratio between the specific region and the region to be located in the target image, as well as the positional information, the bounding box position information of the region to be located is determined.

[0161] Of course, the electronic device may also include a communication interface, a display unit, and an input device, etc., without any specific limitations.

[0162] On the other hand, this application also provides a storage medium storing a computer program, which, when loaded and executed by a processor, is used to implement the image border position detection method described in any of the above embodiments.

[0163] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0164] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0165] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0166] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image frame position detection method, characterized by, Be applied to game live broadcast, including: Cut out a video to be extracted from the received video, the video to be extracted being a video of a preset time length of an intermediate section of the video; extract a target image from the video to be extracted; Respectively extract feature information of a predetermined region of the target image containing a specific region, and feature information of a specific region template; the predetermined region is smaller in size than the entire target image, the specific region has fixed features and has a corresponding positional proportional relationship with a region to be positioned in the target image, the region to be positioned is a partial region of the target image, the specific region is a partial region of the region to be positioned, the specific region template has the same features or the same fixed features as the specific region, when the specific region belongs to a region that does not change, the specific region template is the specific region, when the specific region has a changeable region, the specific region template is a specific region image with the clearest fixed features in multiple images; when the target image is a game live image, the specific region is a mini-map region, the specific region template is a mini-map region template, the fixed features in the specific region include roads in the mini-map, and the region to be positioned is a game region; Match the specific region template with the feature information of the predetermined region; if the specific region template cannot match the specific region from the predetermined region in the target image, output prompt information indicating that the image bounding box position information detection fails, and output prompt information for reacquiring the target image; if the specific region template matches the specific region from the predetermined region in the target image, count the number of matched feature point pairs; when the number is greater than a preset threshold, use the matched feature point pair information to obtain the position information of the specific region in the target image; According to the positional proportional relationship and the position information, obtain the initial bounding box position information of the region to be positioned, each target image in which the specific region is detected corresponding to an initial bounding box position information; Use an outlier rejection algorithm to reject outlier initial bounding box position information in the initial bounding box position information, and take the mean of the rejected initial bounding box position information to obtain the bounding box position information of the region to be positioned; The initial bounding box position information and the bounding box position information of the region to be positioned both include the horizontal axis coordinate of a first vertex, and the process of processing the initial bounding box position information using the outlier rejection algorithm to obtain the horizontal axis coordinate of the first vertex in the bounding box position information of the region to be positioned includes: The frequency of each coordinate value in the set of horizontal axis coordinate values of the first vertex in all initial bounding box position information is counted, the coordinate values are sorted in descending order of frequency, and a set of candidate coordinate values is obtained; the coordinate values in the set of candidate coordinate values are sequentially removed in descending order of frequency, and the variance of the remaining candidate values is calculated after removing each candidate value; when the variance is higher than a threshold value, the next candidate value is continuously removed until the variance of the remaining candidate values is lower than the threshold value, and then the removal is stopped; the mean value of all remaining candidate values is calculated as the horizontal axis coordinate of the first vertex in the bounding box position information of the region to be positioned; the video is labeled with the bounding box position information of the region to be positioned, so as to realize labeling of the bounding box position information of the region to be positioned in the target image; according to the labeling, the region to be positioned in the uploaded video of the same type as the video is extracted, and the picture content irrelevant to the region to be positioned is removed.

2. The image frame position detection method according to claim 1, characterized by, The matching of the specific region template and the feature information of the predetermined region includes: The specific region template is matched with the feature information of the predetermined region by using the nearest neighbor method.

3. The image frame position detection method according to claim 1, wherein The position information of the specific region in the target image is obtained by using the matched feature point pair information, including: Solving the homography matrix of the matched feature point pair information; Using the homography matrix to perform projection transformation on the position of the specific region in the target image to obtain the position information of the specific region in the target image.

4. An image frame position detection apparatus characterized by comprising: Applied to game live broadcast, including: An acquisition module is configured to extract a video to be extracted from a received video, the video to be extracted being a video of a preset time length in a middle section of the video; and extract a target image from the video to be extracted; A specific region positioning module is configured to extract feature information of a predetermined region containing a specific region in the target image and feature information of a specific region template; the predetermined region has a size smaller than that of the target image as a whole, the specific region has fixed features and has a corresponding position proportional relationship with a region to be positioned in the target image, the region to be positioned is a partial region of the target image, the specific region is a partial region of the region to be positioned, the specific region template has the same features or the same fixed features as the specific region, when the specific region belongs to a region without any changes, the specific region template is the specific region, and when the specific region has changeable regions, the specific region template is a specific region image with the clearest fixed features in multiple images; when the target image is a game live broadcast image, the specific region is a mini-map region, the specific region template is a mini-map region template, the fixed features in the specific region include roads in the mini-map, and the region to be positioned is a game region. The specific region positioning module is further configured to match the specific region template with the feature information of the predetermined region; if the specific region template fails to match the specific region from the predetermined region in the target image, a prompt message indicating failure of image border position information detection is output, and a prompt message indicating re-acquiring the target image is output; if the specific region template matches the specific region from the predetermined region in the target image, the number of matched feature point pairs is counted; when the number is greater than a preset threshold, the position information of the specific region in the target image is obtained using the matched feature point pair information; The image border localization module is used to obtain the initial border position information of the region to be located based on the position ratio relationship and the position information. Each target image that detects the specific region corresponds to an initial border position information. The module uses an outlier removal algorithm to remove outlier initial border position information from the initial border position information and takes the average value of the removed initial border position information to obtain the border position information of the region to be located. The initial border position information and the border position information of the region to be located both include the horizontal axis coordinate of the first vertex. The process of processing the initial border position information using the outlier removal algorithm to obtain the horizontal axis coordinate of the first vertex in the border position information of the region to be located includes: The frequency of each coordinate value in the set of horizontal axis coordinates of the first vertex in all initial border position information is counted, and the coordinate values ​​are sorted from high to low frequency to obtain a set of candidate coordinate values. Coordinate values ​​in the set of candidate coordinate values ​​are eliminated sequentially from high to low frequency. For each eliminated candidate value, the variance of the remaining candidate values ​​is calculated. When the variance is higher than a threshold, the next candidate value is eliminated until the variance of the remaining candidate values ​​is lower than the threshold, at which point elimination stops. The mean of all remaining candidate values ​​is calculated and used as the horizontal axis coordinate of the first vertex in the border position information of the region to be located. A module for performing the following steps: adding a border position information label to the video to label the border position information of the region to be located in the target image; based on the label, extracting the region to be located from uploaded videos of the same type as the video, and removing image content unrelated to the region to be located.

5. The image frame position detection apparatus according to claim 4, characterized by The specific area positioning module is also used for: The nearest neighbor method is used to match the specific region template with the feature information of the predetermined region.

6. The image frame position detection apparatus according to claim 4, characterized by The specific area positioning module is also used for: Solve for the homography matrix of the matched feature point pairs; The homography matrix is ​​used to perform a projection transformation on the location of a specific region in the target image to obtain the location information of the specific region in the target image.

7. An electronic device, comprising: include: Memory, used to store computer programs; A processor is configured to implement the image border position detection method as described in any one of claims 1 to 3 when executing the computer program.

8. A storage medium, characterized by The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the image border position detection method as described in any one of claims 1 to 3.

Images