Visual resource detection method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a visual resource detection method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: obtaining a first image frame corresponding to a first moment from a first video comprising a first visual resource; a second image frame corresponding to the first moment is obtained from a second video comprising a second visual resource, at least one third image frame adjacent to the second image frame is obtained, and the first visual resource is obtained by updating visual elements in the second visual resource; splicing the first image frame, the second image frame and the at least one third image frame into an input image according to a channel; performing feature extraction on the input image to obtain an image feature vector; and obtaining a version update detection result of the first visual resource and the second visual resource based on the image feature vector. According to the invention, the accuracy of visual resource detection can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting visual resources. Background Technology

[0002] When software or web pages are updated, visual resources between the old and new versions often become abnormal due to the update. Therefore, it is necessary to check the visual resources of different versions to find the abnormal visual resources so that they can be corrected in a timely manner.

[0003] Related technologies typically sample image frames from different versions of visual resources at the same time and then compare the differences between the image frames at the same time to obtain the detection result, which can lead to inaccurate detection. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting visual resources, which can improve the accuracy of visual resource detection.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for detecting visual resources, the method comprising:

[0007] Obtain the first image frame corresponding to the first moment from the first video including the first visual resource;

[0008] A second image frame corresponding to the first moment is obtained from a second video including a second visual resource, and at least one third image frame adjacent to the second image frame is obtained, wherein the first visual resource is obtained by updating the visual elements in the second visual resource;

[0009] The first image frame, the second image frame, and the at least one third image frame are stitched together according to channels to form an input image;

[0010] Feature extraction is performed on the input image to obtain an image feature vector;

[0011] The version update detection results of the first visual resource and the second visual resource are obtained based on the image feature vector, wherein the version update detection results characterize whether the update of the first visual resource relative to the second visual resource is normal.

[0012] This application provides a visual resource detection device, the device comprising:

[0013] The first acquisition module is configured to acquire a first image frame corresponding to a first moment from a first video including a first visual resource;

[0014] The second obtaining module is configured to obtain a second image frame corresponding to the first moment from a second video including a second visual resource, and to obtain at least one third image frame adjacent to the second image frame, wherein the first visual resource is obtained by updating the visual elements in the second visual resource;

[0015] The image stitching module is used to stitch the first image frame, the second image frame, and the at least one third image frame into an input image according to channels;

[0016] The feature extraction module is used to extract features from the input image to obtain an image feature vector;

[0017] The result acquisition module is used to obtain the version update detection result of the first visual resource and the second visual resource based on the image feature vector, wherein the version update detection result indicates whether the update of the first visual resource relative to the second visual resource is normal.

[0018] This application provides an electronic device, the electronic device comprising:

[0019] Memory is used to store executable instructions for a computer;

[0020] The processor, when executing computer-executable instructions stored in the memory, implements the visual resource detection method provided in the embodiments of this application.

[0021] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the visual resource detection method provided in this application when executed by a processor.

[0022] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the visual resource detection method provided in this application.

[0023] The embodiments of this application have the following beneficial effects:

[0024] By stitching together the first image frame of the new version's visual resources (i.e., the first visual resource) at the first moment, the second image frame of the historical version's visual resources (i.e., the second visual resource) at the first moment, and the third image frame adjacent to the second image frame, an input image is obtained. This allows the acquisition of version update detection results to reference the third image frame associated with the second image frame, thereby mining image frames from the historical version that will affect the version update detection results from multiple perspectives. By utilizing the correlation between adjacent image frames of the historical version and the first image frame of the new version, the accuracy of detection is improved. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the architecture of the visual resource detection system 100 provided in an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application;

[0027] Figure 3A This is a schematic diagram of the first process of the visual resource detection method provided in the embodiments of this application;

[0028] Figure 3B This is a schematic diagram of the second process of the visual resource detection method provided in the embodiments of this application;

[0029] Figure 3C This is a schematic diagram of the third process of the visual resource detection method provided in the embodiments of this application;

[0030] Figure 3D This is a schematic diagram of the fourth process of the visual resource detection method provided in the embodiments of this application;

[0031] Figure 3E This is a schematic diagram of the fifth process of the visual resource detection method provided in the embodiments of this application;

[0032] Figure 3F This is a schematic diagram of the sixth process of the visual resource detection method provided in the embodiments of this application;

[0033] Figure 3G This is a schematic diagram of the seventh process of the visual resource detection method provided in the embodiments of this application;

[0034] Figure 3H This is a schematic diagram of the eighth process of the visual resource detection method provided in the embodiments of this application;

[0035] Figure 3I This is a schematic diagram of the ninth process of the visual resource detection method provided in the embodiments of this application;

[0036] Figure 3JThis is a schematic diagram of the tenth process of the visual resource detection method provided in the embodiments of this application;

[0037] Figure 3K This is a schematic diagram of the eleventh step of the visual resource detection method provided in the embodiments of this application;

[0038] Figure 3L This is a schematic diagram of the twelfth step of the visual resource detection method provided in the embodiments of this application;

[0039] Figure 3M This is a schematic diagram of the thirteenth step of the visual resource detection method provided in the embodiments of this application;

[0040] Figure 3N This is a schematic diagram of the fourteenth step of the visual resource detection method provided in the embodiments of this application;

[0041] Figure 4 This is a schematic diagram of the application mask provided in the embodiments of this application;

[0042] Figure 5 This is a schematic diagram illustrating the principle of feature extraction provided in the embodiments of this application;

[0043] Figure 6 This is a schematic diagram of the Transformer encoder provided in the embodiments of this application;

[0044] Figure 7 This is a schematic diagram illustrating the principle of training a machine learning model according to an embodiment of this application;

[0045] Figure 8 This is a schematic diagram of the determination result provided in the embodiments of this application;

[0046] Figure 9 This is a schematic diagram illustrating the principle of the visual resource detection method provided in the embodiments of this application.

[0047] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] In the following description, the terms "first / second / third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0051] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0052] Unless otherwise specified, "at least one" as used below refers to one or more cases, and "multiple" can refer to two or more cases.

[0053] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0054] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0055] 1) Visual resources refer to the materials used in videos to design and construct visual elements such as characters, scenes, or props. These resources are usually created by art designers and art teams and play a crucial role in game or video production. Visual resources include character models, environmental backgrounds, virtual props, textures and materials of objects, special effects, and lighting effects.

[0056] 2) Version update detection results indicate whether the version update of visual resources is normal. For cross-version visual resource updates, if the similarity between the first and second visual resources is small, the version update detection result is normal; otherwise, if the similarity is large, the version update detection result is abnormal. For intra-version visual resource updates, if the similarity between the first and second visual resources is small, the version update detection result is abnormal; otherwise, if the similarity is large, the version update detection result is normal.

[0057] 3) A keyframe is a crucial image in a sequence of images, typically used to define an important moment or state in an animation. In video compression and transmission, a keyframe is also called an intra-coded frame (I-frame), which is an independent image that can be reconstructed without referencing other frames.

[0058] 4) A mask is a binary number used to mask or select specific information. In image processing, masks are used to define specific regions in an image, allowing you to select, enhance, or hide certain parts of the image. For example, a grayscale image can be used as a mask to control the transparency of another image.

[0059] 5) A convolutional kernel is a small matrix used to perform convolution operations on an image, consisting of a series of weight values. The main function of a convolutional kernel is to extract features from the input image and transform these features into a higher-level feature representation.

[0060] 6) A forward frame refers to one or more frames in a video stream or image sequence whose timestamp precedes the timestamp of a specific image frame. For example, if the image frames in a video stream are arranged in chronological order as follows: image frame A, image frame B, image frame C, image frame D, and image frame E, and image frame C is selected as the specific image frame, then image frames B and A are the forward frames.

[0061] 7) A backward image frame refers to one or more frames in a video stream or image sequence whose timestamp is located after the timestamp of a specific image frame. For example, if the image frames in a video stream are arranged in chronological order as follows: image frame A, image frame B, image frame C, image frame D, and image frame E, and image frame C is selected as the specific image frame, then image frames D and E are backward image frames.

[0062] 8) Cross-version updates refer to the method of upgrading software, games, and other products directly from one version to another. Cross-version updates are usually accompanied by significant functional changes or interface redesigns, making them more noticeable than minor updates. For example, the version number format is vx.y, where x represents the major version number and y represents the minor version number. Increasing the value of x indicates a cross-version update. Cross-version updates often involve the replacement, addition, or deletion of visual elements. For example, if the second visual element is a cat, the first visual element could be a dog instead of a cat. The following are some characteristics of cross-version updates:

[0063] Significant feature changes: Cross-version updates typically introduce new features, modify existing features, or completely redesign some features. This type of update may change user habits, requiring users to readjust.

[0064] Interface redesign: Cross-version updates may be accompanied by an interface redesign, including UI layout adjustments, style changes, and color scheme updates. Such changes not only affect the user experience but also enhance the product's visual appeal.

[0065] In general, cross-version updates are a large-scale update method, often accompanied by significant changes to functionality and the user interface. Through cross-version updates, products can be continuously optimized and improved to adapt to changing market demands and enhance the user experience.

[0066] 9) Intra-version updates, also known as minor version updates or iterative updates, refer to updates made to software, games, and other products within the same major version number. These updates typically do not involve significant functional changes or interface redesigns, but rather focus on bug fixes, performance optimizations, adding minor features, and improving user experience. For example, the version number format is vx.y, where x represents the major version number and y represents the minor version number. If the value of x remains unchanged while the value of y increases, it indicates an intra-version update. Intra-version updates often involve minor changes to visual elements; for example, if the secondary visual element is a cat, the primary visual element might be a change to the cat's fur or facial features. Here are some characteristics of intra-version updates:

[0067] Bug Fixes: The main purpose of this update is to fix bugs, errors, and issues found in the previous version, ensuring the stability and reliability of the product.

[0068] Performance optimization: Updates may include optimizations to product performance, such as increasing running speed, reducing memory usage, and improving response time, to enhance the user experience.

[0069] Feature enhancements: Updates within a version may add some new features or improve existing features, but these changes are usually minor and will not affect users' basic usage habits.

[0070] User interface tweaks: Updates may include tweaks to the user interface, such as improving button layouts, optimizing font sizes, and adjusting color schemes, to enhance visual appeal and ease of use.

[0071] In general, in-version updates are a smaller-scale update method, mainly focusing on bug fixes, performance optimization, and user experience improvements, without significantly impacting the product's core functionality and interface design. Frequent in-version updates allow the product to continuously improve and refine, maintaining its competitiveness.

[0072] When a game is updated, visual resources from both the old and new versions often exhibit anomalies due to the update. Related technologies typically sample image frames from different versions of visual resources at the same time, then manually compare the differences between these frames to arrive at the results. However, when a game contains a large number of visual resources, these detection methods become prohibitively expensive. Furthermore, new versions may introduce configuration changes or bug fixes, meaning the movements of models and virtual characters in the game scene may not align with the current frame of a previous version, but rather with frames preceding or following them. Therefore, relying solely on image frames from the same timeframe can lead to inaccurate detection results.

[0073] Based on the above analysis, the applicant found that the visual resource detection methods of related technologies cannot efficiently and accurately detect visual resources after version updates. In order to address the above problems, this application provides a visual resource detection method that can improve the accuracy of visual resource detection.

[0074] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting visual resources, which can improve the accuracy of visual resource detection. The following describes exemplary applications of the electronic device provided in this application. The electronic device provided in this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or it can be implemented as a server. The following will describe exemplary applications when the electronic device is implemented as a server.

[0075] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the visual resource detection system 100 provided in the embodiments of this application. In order to support a visual resource detection application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0076] Server 200 combines the first image frame at the first moment, the second image frame at the first moment, and at least one third image frame adjacent to the second image frame to obtain the version update detection result of the first visual resource and the second visual resource, and sends the version update detection result to terminal 400 for display on human-computer interaction interface 410.

[0077] Taking the application of the visual resource detection method to server 200 as an example, during the process of running a new version of the game on terminal 400, server 200 can obtain the first image frame during the display of visual resources and compare it with the second image frame obtained from the old version at the same time to generate a version update detection result, such as normal update or abnormal update. The version update detection result is sent to terminal 400 and displayed on human-computer interaction interface 410.

[0078] Taking the visual resource detection method applied to terminal 400 as an example, terminal 400 locally obtains a first image frame containing visual resources in the new version of the game, and obtains a second image frame containing visual resources at the same time from the old version of the game. By comparing the first image frame and the second image frame, a version update detection result is generated, such as normal update or abnormal update, and displayed on the human-computer interaction interface 410.

[0079] Taking the method of visual resource detection performed by the terminal and server as an example, the terminal 400 obtains the first image frame containing visual resources in the new version of the game and obtains the second image frame containing visual resources at the same time from the old version of the game. The first image frame and the second image frame are sent to the server 200. The server compares the two image frames, generates the version update detection result, and sends the result to the terminal 400 for display on the human-computer interaction interface 410.

[0080] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0081] The embodiments of this application can be implemented using artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0082] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0083] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. The electronic device 500 can be the server 200 or the terminal 400 described above. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0084] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0085] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0086] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0087] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0088] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0089] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0090] The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0091] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0092] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0093] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An image processing apparatus 555 stored in memory 550 is shown. This apparatus can be software in the form of programs and plug-ins, and includes the following software modules: a first acquisition module 5551, a second acquisition module 5552, an image stitching module 5553, a feature extraction module 5554, and a result acquisition module 5555. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0094] The following describes the visual resource detection method provided in this application embodiment, taking the server as the execution subject as an example, and in conjunction with the exemplary application and implementation of the server provided in the embodiments of this application.

[0095] See Figure 3A , Figure 3A This is a schematic diagram of the first process of the visual resource detection method provided in this application embodiment, with the server as the main body, and combining... Figure 3A The steps shown are explained.

[0096] In step 101, a first image frame corresponding to a first moment is obtained from a first video including a first visual resource.

[0097] For example, taking a game scene as an example, the first video could be the game's opening or cutscene animation, or real-time footage from the game. Visual resources could include game character models, environmental backgrounds, virtual props, object textures and materials, special effects, and lighting effects.

[0098] For example, the "first moment" can be any moment within the time period during which visual resources are displayed in the first video. The "first moment" is not specific; it can be one or more moments, such as multiple moments at fixed intervals (30 seconds or 2 minutes) after the visual resources are displayed. The following explanation uses a single "first moment" as an example. The time period for displaying the visual resources can be obtained from the metadata or attribute data of the first video itself. During the production stage of the first video, the visual resources to be displayed and their corresponding time periods are pre-defined. The first image frame is an image frame containing the visual resources captured at the first moment, and the number of first image frames is the same as the number of first moments.

[0099] In step 102, a second image frame corresponding to the first moment is obtained from the second video including the second visual resource, and at least one third image frame adjacent to the second image frame is obtained, wherein the first visual resource is obtained by updating the visual elements in the second visual resource.

[0100] In some embodiments, the first visual resource is obtained by updating the second visual resource. For example, if the version number of the second visual resource is v1, then the version number of the first visual resource can be v2.

[0101] In some embodiments, at least one third image frame adjacent to the second image frame may be an image frame directly adjacent to the second image frame, such as the previous frame or the next frame.

[0102] In other embodiments, at least one third image frame adjacent to the second image frame may also be an image frame indirectly adjacent to the second image frame, for example, an image frame whose number of image frames between it and the second image frame is less than a frame number threshold, and whose type of visual resource is the same as that of the visual resource in the second image frame.

[0103] For example, in the multimedia container format (MPEG-4 Part 14, MP4), the third image frame can be any type of frame, such as an I-frame, a predicted frame (P-frame), or a bidirectionally predicted frame (B-frame).

[0104] For example, if the frame number threshold is 5, and the type of visual resource in the second image frame is a model of virtual object A, then if the number of image frames between an image frame and the second image frame is 3, and also includes the model of virtual object A, then that image frame is taken as the third image frame.

[0105] In some embodiments, see Figure 3B , Figure 3B This is a schematic diagram of the second process of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 102, "obtaining at least one third image frame adjacent to the second image frame," can be achieved through... Figure 3B Steps 1021A to 1022A are implemented, and the details are explained below.

[0106] In step 1021A, the image group in which the second image frame is located is determined from the second video.

[0107] In some embodiments, taking MP4 type video as an example, the type of the second image frame may include I-frame, P-frame and B-frame, and the group of pictures (GOP) to which the second image frame belongs is determined, wherein each group of pictures starts with an I-frame.

[0108] In step 1022A, at least one image frame in the image group other than the second image frame is designated as at least one third image frame adjacent to the second image frame.

[0109] Exemplarily, if an image group is <I frame, P frame, B frame, I frame, P frame, B frame>, where the first I frame is the first frame of the image group, including the complete information of the image, not depending on other frames, with a frame number of 0, and the image shows "Scene A"; the first P frame includes partial information that has changed compared to the I frame, with a frame number of 1, and the image shows "The actions of the characters in Scene A"; the first B frame is encoded using the frames before and after, with a frame number of 2, and the image shows "The characters in Scene A continue their actions while showing a static image of Scene B"; the second I frame provides a new reference point, with a frame number of 3, and the image shows "New actions in Scene B"; the second P frame has a frame number of 4, and the image shows "The actions of the characters in Scene B"; the second B frame has a frame number of 5, and the image shows "The characters in Scene B continue their actions while showing a static image of Scene A". If the second image frame is the first P frame, i.e., the frame with a frame number of 1, then at least one third image frame can be the first B frame, the second P frame, or the second B frame.

[0110] In the embodiments of the present application, by determining the image group where the second image frame is located and selecting the third image frame from the image group, the image frames within the same image group are temporally continuous, making the selected third image frame visually coherent with the second image frame, avoiding noise caused by the actual change of visual resources when the image frames are far apart, and ensuring the accuracy of the detection result.

[0111] In some embodiments, refer to Figure 3C , Figure 3C which is the third process schematic diagram of the detection method for visual resources provided by the embodiments of the present application. Figure 3A Step 102 "Obtain at least one third image frame adjacent to the second image frame" of Figure 3C can also be implemented through steps 1021B to 1023B of

[0112] In step 1021B, select at least one forward image frame located before the second image frame.

[0113] In some embodiments, at least one forward image frame can be a forward image frame directly adjacent to the second image frame or a forward image frame indirectly adjacent to the second image frame. Among them, the timestamp of the forward image frame is before the timestamp of the second image frame, that is, the moment corresponding to the timestamp of the forward image frame is before the moment corresponding to the timestamp of the second image frame.

[0114] In step 1022B, select at least one backward image frame located after the second image frame, where the number of at least one forward image frame is the same as the number of at least one backward image frame.

[0115] In some embodiments, at least one backward image frame may be a backward image frame directly adjacent to the second image frame or a backward image frame indirectly adjacent to the second image frame, wherein the timestamp of the backward image frame is located after the timestamp of the second image frame, and the number of at least one forward image frame is the same as the number of at least one backward image frame.

[0116] For example, if the number of at least one forward timestamp is 2, then the number of at least one backward timestamp is also 2.

[0117] This application embodiment selects the same number of forward and backward image frames to ensure that the differences between image frames can be compared in a balanced manner within the time period before and after the second image frame. This avoids bias in the detection results due to too many forward or backward image frames, and improves the accuracy and impartiality of the detection.

[0118] In step 1023B, at least one forward image frame and at least one backward image frame are traversed. The average gray value of any forward image frame and any backward image frame currently being traversed is determined according to the channel. The average gray value of each channel is then stitched together to form the third image frame.

[0119] For example, if the number of forward image frames is 1, and the gray values ​​of the red, green and blue channels are 50, 60 and 70 respectively, and the number of backward image frames is 1, and the gray values ​​of the red, green and blue channels are 70, 80 and 90 respectively, then the average gray values ​​of the three channels are 60, 70 and 80 respectively. The third image frame is constructed according to the average gray values ​​of the three channels.

[0120] This application embodiment selects multiple image frames forward and backward from the second image frame, determines the average gray value according to the channel, and concatenates the average gray value of each channel into a third image frame. This allows the third image frame to fuse the features of multiple image frames adjacent to the second image frame, ensuring that the fusion features of multiple image frames adjacent to the second image frame are fully learned during the prediction process. This enhances the correlation between the prediction result and the adjacent image frames of the second image frame, and improves the prediction accuracy.

[0121] In some embodiments, step 102, “obtaining at least one third image frame adjacent to the second image frame”, can also be implemented by performing at least one of the following processes: Method 1, obtaining at least one forward image frame from the second video as the third image frame, wherein the timestamp of the forward image frame is located before the timestamp of the second image frame; Method 2, obtaining at least one backward image frame from the second video as the third image frame, wherein the timestamp of the backward image frame is located after the timestamp of the second image frame.

[0122] In some embodiments, only the forward or backward image frames of the second image frame can be obtained as the third image frame, or both directions can be obtained as the third image frame.

[0123] In some embodiments, the third image frame may be a keyframe among a plurality of image frames adjacent to the second image frame.

[0124] As an example, image processing techniques can be used to analyze the content of multiple image frames adjacent to the second image frame, identify important changes in the image frames, such as the appearance, disappearance, or motion trajectory of objects, and when these changes are detected, the image frame that has changed is marked as a keyframe and used as the third image frame.

[0125] As an example, a machine learning model can be trained to identify keyframes. For instance, multiple image frame samples are acquired, and a true class label is added to each image frame sample. The true class label indicates whether the image frame sample is a keyframe. The machine learning model is called based on the image frame samples to obtain the predicted label. The difference between the true label and the predicted label is calculated as the loss value. The parameters of the machine learning model are updated based on the loss value to obtain the trained machine learning model. Multiple image frames adjacent to the second image frame are taken as input and sequentially fed into the trained machine learning model to obtain the predicted label of each image frame. The image frame whose predicted label indicates that it is a keyframe is taken as the third image frame.

[0126] The embodiments of this application simultaneously consider the preceding and following image frames of the second image frame, enhancing the correlation between the prediction result and the adjacent image frames of the second image frame, and improving the prediction accuracy.

[0127] In some embodiments, the number of at least one forward image frame and at least one backward image frame are both preset numbers, see [link to relevant documentation]. Figure 3D , Figure 3D This is a schematic diagram of the fourth process of the visual resource detection method provided in this application embodiment. Before step 102, the following steps are executed: Figure 3D Steps 201 to 203 are explained in detail below.

[0128] In step 201, the first frame rate of the first video and the second frame rate of the second video are obtained.

[0129] In some embodiments, video analysis tools, such as Video LAN Client Media Player or the open-source audio-visual multimedia processing framework (Fast Forward MovingPicture Experts Group, FFMPEG), can be used to obtain the frame rate of the video through command-line parameters or API functions of the video analysis tools.

[0130] In step 202, the larger frame rate between the first frame rate and the second frame rate is determined.

[0131] For example, if the first frame rate is 20 frames per second (FPS) and the second frame rate is 10 frames per second, then the first frame rate is the larger frame rate.

[0132] In step 203, the preset quantity is based on the product of a larger frame rate and a preset coefficient.

[0133] For example, if the larger frame rate is the first frame rate of 20 frames per second and the preset coefficient is 0.2, then the preset number is 4, that is, the number of forward image frames and backward image frames is 4 each.

[0134] In some embodiments, a preset coefficient is a parameter used to adjust the frame rate when comparing old and new image frames. The value of the preset coefficient is less than 1.

[0135] The embodiments of this application use a preset coefficient with a value less than 1, so that the actual frame rate used will be lower than the original frame rate. By reducing the frame rate, the differences between image frames can be observed and compared more carefully.

[0136] In some embodiments, the number of forward image frames and the number of backward image frames can be preset fixed values, such as 2.

[0137] The embodiments of this application determine the number of obtained third image frames by frame rate, ensuring that the number of obtained third image frames is positively correlated with the frame rate. This allows for the acquisition of more third image frames at high frame rates and fewer third image frames at low frame rates. This avoids the problem of inaccurate detection results due to insufficient third image frames obtained because the frame rate is too high, which fails to fully reflect the complete image changes adjacent to the second image frame, or inaccurate detection results due to excessive noise caused by too many third image frames obtained because the frame rate is too low. This improves the accuracy of prediction.

[0138] See also Figure 3A In step 103, the first image frame, the second image frame, and at least one third image frame are stitched together according to channels to form an input image.

[0139] As an example, for each channel, the data of the first image frame, the second image frame, and at least one third image frame in the corresponding channel are stitched together. If the number of at least one third image frame is 1, taking the R channel as an example, the data of the first image frame, the second image frame, and at least one third image frame in the R channel are denoted as R1, R2, and R3 respectively, and the stitching results in R1-R2-R3, which is used as the stitching result of the R channel.

[0140] Similarly, taking the G channel as an example, the data of the first image frame, the second image frame, and at least one third image frame in the G channel are denoted as G1, G2, and G3, respectively. These are then concatenated to obtain G1-G2-G3, which serves as the concatenation result for the G channel. Taking the B channel as an example, the data of the first image frame, the second image frame, and at least one third image frame in the B channel are denoted as B1, B2, and B3, respectively. These are then concatenated to obtain B1-B2-B3, which serves as the concatenation result for the B channel. The final input image is (R1-R2-R3, G1-G2-G3, B1-B2-B3).

[0141] For example, continuing from the previous example, if the number of third image frames is 1, and the size of the first, second, and third image frames is 32x32 pixels, with 3 channels (RGB format), taking the R channel as an example, the data of the first image frame in the R channel is R1: [255, 0, 0], the data of the second image frame in the R channel is R2: [0, 255, 0], and the data of the third image frame in the R channel is R3: [0, 0, 255]. The concatenation result of the R channel is [[255, 0, 0][0, 255, 0][0, 0, 255]]. Taking the G channel as an example, the data of the first image frame in the G channel is G1: [0, 255, 0, 0, 0, 255]]. The data of the second image frame in the G channel is G2: [255, 0, 0], the data of the third image frame in the G channel is G3: [0, 0, 255], and the splicing result of the G channel is [[0, 255, 0][255, 0, 0][0, 0, 255]]. Taking the B channel as an example, the data of the first image frame in the B channel is B1: [0, 0, 255], the data of the second image frame in the B channel is B2: [255, 0, 0], the data of the third image frame in the B channel is B3: [0, 255, 0], and the splicing result of the B channel is [[0, 0, 255][255, 0, 0][0, 255, 0]]. The input image is: {[[255,0,0][0,255,0][0,0,255]]; [[0,255,0][255,0,0][0,0,255]]; [[0,0,255][255,0,0][0,255,0]]}.

[0142] In some embodiments, see Figure 3E , Figure 3E This is a schematic diagram of the fifth process of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 103 can be achieved through Figure 3E Steps 1031A to 1034A are implemented, and the details are explained below.

[0143] In step 1031A, target detection is performed on the second image frame and the third image frame to obtain the imaging region of the second visual resource in the second image frame and the third image frame.

[0144] In some embodiments, the imaging region of the second visual resource in the second and third image frames can be obtained through object detection algorithms. For example, methods based on traditional image processing, such as edge detection and contour extraction; machine learning methods, such as Support Vector Machine (SVM) and Random Forest; and deep learning methods, such as Convolutional Neural Networks (CNN), YouOnly Look Once (YOLO), or Single Shot MultiBox Detector (SSD).

[0145] For example, using the YOLO model, firstly, the second and third image frames are preprocessed, such as resizing the images to a specified size, converting them to floating-point format, and normalizing them to the [0, 1] interval. Secondly, the preprocessed image frames are input into the pre-trained YOLO model, resulting in an array containing multiple predictions for the second and third image frames, respectively. Each prediction includes the coordinates of the bounding box of the image frame. Then, non-maximum suppression (NMS) is used to merge overlapping bounding boxes, retaining the highest-scoring bounding box and removing redundant bounding boxes with lower scores. Finally, the region within the highest-scoring bounding box in each array is used as the imaging region of the second visual resource in the second and third image frames.

[0146] In step 1032A, a mask is generated based on the area outside the imaging area.

[0147] In some embodiments, a mask is typically used to specify a region of an image so that certain operations can be performed on that region without affecting other parts of the image.

[0148] For example, see Figure 4 , Figure 4 This is a schematic diagram of the application mask provided in an embodiment of this application. Figure 4First, two masks with the same size as the second image frame 401 and the third image frame 406 are created, namely the mask 403 of the second image frame and the mask 408 of the third image frame. All pixel values ​​of the imaging region 402 of the second image frame 401 in the corresponding part 404 of the mask 403 are set to 1. All pixel values ​​of the imaging region 407 of the third image frame 406 in the corresponding part 409 of the mask 408 are set to 1. The pixel values ​​of the area outside the corresponding part 404 of the imaging region 402 in the mask 403 are set to 0. The pixel values ​​of the area outside the corresponding part 409 of the imaging region 407 in the mask 408 are set to 0.

[0149] In step 1033A, masks are applied to the second image frame and the third image frame respectively to obtain the masked second image frame and the third image frame.

[0150] In some embodiments, the generated mask is bitwise ANDed with the second and third image frames respectively, retaining the portion of the image with a mask value of 1 and removing the portion with a mask value of 0.

[0151] For example, see Figure 4 The second image frame 401 is ANDed with its mask 403, retaining the portions with mask 1s and removing the portions with mask 0s, resulting in the second image frame 405 after applying the mask. Similarly, the third image frame 406 is ANDed with its mask 408, retaining the portions with mask 1s and removing the portions with mask 0s, resulting in the third image frame 411 after applying the mask.

[0152] For example, if the pixel values ​​of the second image frame are: {[255, 255, 255][0, 0, 255][255, 0, 0]#red}, and the pixel values ​​of the mask are: {[1, 1, 1][0, 0, 1][1, 0, 1]}, then the pixel values ​​of the second image frame after applying the mask are: {[255, 255, 255][0, 0, 0][255, 0, 0]}. The white and red areas in the original second image frame are preserved, while the blue areas are removed because the corresponding pixel values ​​for the white and red areas in the mask are 1, while the mask pixel values ​​for the blue areas are 0.

[0153] In step 1034A, the first image frame, the second image frame after applying masks, and the third image frame are stitched together according to channels to form the input image.

[0154] For example, referring to the example of step 103 above, the first image frame, the second image frame after applying masks, and the third image frame are stitched together according to different channels to obtain the input image.

[0155] In this embodiment, the target detection algorithm is used to mask the second and third image frames before stitching them together with the first image frame. This method retains only the visual resources in the second and third image frames and removes noise introduced by areas other than the imaging area in the second and third image frames. This allows the prediction process to focus more on the imaging area and improves the accuracy of the prediction.

[0156] In some embodiments, see Figure 3F , Figure 3F This is a schematic diagram of the sixth process of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 103 can also be done through Figure 3F Steps 1031B to 1032B are implemented, and the details are explained below.

[0157] In step 1031B, the frame difference is determined based on the first image frame, the second image frame, and at least one third image frame.

[0158] In some embodiments, see Figure 3G , Figure 3G This is a schematic diagram of the seventh process of the visual resource detection method provided in the embodiments of this application. Figure 3F Step 1031B can be achieved through Figure 3G Steps 1031B1 to 1031B5 are implemented, and the details are explained below.

[0159] In step 1031B1, the first grayscale value of the pixel in the first image frame and the second grayscale value of the pixel in the second image frame that is at the same position as the pixel in the first image frame are obtained.

[0160] In some embodiments, a first image frame and a second image frame are loaded, the sizes of the two image frames are obtained, an array of the same size as the first image frame is created to store the results, each pixel in the first image is traversed, the gray value of the current pixel in the first image is obtained, the gray value of the pixel in the second image that is at the same position as the pixel in the first image is obtained, and the two gray values ​​are stored in the result array as a tuple.

[0161] For example, the pixel values ​​of the first image frame are: {[100, 150, 200][120, 170, 220][140, 190, 240]}, and the pixel values ​​of the second image frame are: {[200, 150, 100][220, 170, 80][240, 190, 60]}. If the first grayscale value of a pixel in the first image frame is 200, then the second grayscale value of a pixel in the second image frame that is at the same position as a pixel in the first image frame is 100.

[0162] In step 1031B2, a first difference between the first grayscale value and the second grayscale value is determined.

[0163] For example, following the example of step 1031B1 above, if the first grayscale value of a pixel in the first image frame is 200, then the second grayscale value of a pixel in the second image frame that is at the same position as the pixel in the first image frame is 100, and the first difference is 100. The first differences corresponding to all pixels are [100, 0, 100, 100, 0, 140, 100, 0, 180].

[0164] In step 1031B3, the third grayscale value of at least one pixel in a third image frame that is at the same position as a pixel in the first image frame is obtained.

[0165] For example, following the example of step 1031B1 above, the pixel values ​​of the third image are {[60, 150, 190][100, 160, 210][40, 90, 240]}. If the first gray value of the pixel in the first image frame is 200, then the second gray value of the pixel in the second image frame that is at the same position as the pixel in the first image frame is 190.

[0166] In step 1031B4, a second difference between the first gray value and the third gray value is determined.

[0167] For example, following the example of step 1031B3 above, if the first grayscale value of a pixel in the first image frame is 200, then the second grayscale value of a pixel in the second image frame that is at the same position as the pixel in the first image frame is 190, and the second difference is 10. The second differences corresponding to all pixels are [40, 0, 10, 20, 10, 10, 100, 100, 0].

[0168] In step 1031B5, a minimum difference is determined based on the first difference and the second difference, and is used as the frame difference.

[0169] In some embodiments, a first average value of the first difference corresponding to all pixels and a second average value of the second difference corresponding to all pixels are determined, and the image composed of multiple differences corresponding to the smallest average value among the first average value and the second average value is used as the frame difference.

[0170] For example, following the examples from steps 1031B1 to 1031B4 above, the first average value is (100+0+100+100+0+140+100+0+180) / 9 = 720 / 9 = 80, and the second average value is (40+0+10+20+10+10+100+100+0) / 9 = 290 / 9. The second average value is less than the first average value. Therefore, [40, 0, 10, 20, 10, 10, 100, 100, 0] is taken as the frame difference.

[0171] See also Figure 3F In step 1032B, the first image frame, the second image frame, at least one third image frame, and the frame difference are stitched together to form the input image.

[0172] For example, referring to the example of step 103 above, the first image frame, the second image frame, at least one third image frame, and the frame difference are stitched together according to different methods to obtain the input image.

[0173] This application embodiment stitches together a first image frame, a second image frame, at least one third image frame, and frame differences into an input image. This allows for a more direct learning of the differences between image frames during visual resource detection, improving prediction efficiency. Furthermore, by fusing the first image frame with the frame differences of the second image frame and at least one third image frame into the input image, the impact of image differences is fully considered during prediction, thus improving prediction accuracy.

[0174] See also Figure 3A In step 104, feature extraction is performed on the input image to obtain the image feature vector.

[0175] In some embodiments, see Figure 3H , Figure 3H This is a schematic diagram of the eighth process of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 104 can be achieved through Figure 3H Steps 1041 to 1044 are implemented, and the details are explained below.

[0176] In step 1041, feature extraction is performed based on the input image to obtain an initial feature map.

[0177] In some embodiments, a deep residual network (Residual Network50, Resnet50) can be used as a feature extractor. If the input image is a multi-channel image, the degree of attention paid to each channel and the degree of attention paid to different regions in the channel during feature extraction can be different. Therefore, a convolutional block attention module (CBAM) can also be used to learn the contribution of different channels and different spaces to the generated features.

[0178] See Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of feature extraction provided in this application embodiment. ResNet50 contains a total of five convolutional layers. CBAM is added after each convolutional layer except the first one. This is because the first convolutional layer is responsible for learning low-level features, which are important throughout the image. Subsequent convolutional layers need to learn high-level features, which often only appear in certain regions or channels of the image. Therefore, CBAM is used after the last four convolutional layers of ResNet50, but not after the first layer.

[0179] This application embodiment adds a convolutional attention mechanism after the convolutional layer, which enhances the retention rate of important features when extracting input images, so that the extracted initial feature map can better represent the input image, improve the accuracy of prediction, and at the same time, feature extraction also reduces the computational cost of subsequent processing.

[0180] In some embodiments, see Figure 3I , Figure 3I This is a schematic diagram of the ninth process of the visual resource detection method provided in the embodiments of this application. Figure 3H Step 1041 can be achieved through Figure 3I Steps 10411 to 10412 are implemented, and the details are explained below.

[0181] In step 10411, the data contained in each channel of the input image are subjected to depthwise convolution to obtain the scalar product corresponding to each channel.

[0182] In some embodiments, depthwise convolution (or channel-wise convolution) is a component of depthwise separable convolution. In depthwise convolution, one kernel is responsible for one channel, and each channel is convolved by only one kernel. The number of scalar products produced is exactly the same as the number of input channels. A scalar product is the result of the convolution operation, typically existing in the form of a matrix or vector.

[0183] For example, taking an input image with dimensions of 224×224×3 as an example, the data contained in each channel of the input image is convolved by three convolution kernels with a channel count of 1, resulting in three scalar products of 7×7×1.

[0184] In step 10412, the scalar products corresponding to each channel are stacked, and the stacked scalar products are then subjected to pointwise convolution to obtain the initial feature map.

[0185] As an example, pointwise convolution is also a component of depthwise separable convolution. Pointwise convolution weights and combines the scalar products obtained during depthwise convolution in the depth direction to obtain the image features of the image to be detected.

[0186] For example, following the example of step 10411 above, the obtained three 7×7×1 scalar products are stacked to obtain 7×7×3 image data; the 7×7×3 image data is then convolved with 1280 convolution kernels with 3 channels each to obtain an initial feature map of 7×7×1280.

[0187] In this embodiment, the input image is actually transformed only once (depthwise convolution processing), and then the transformed image is simply stretched to 1280 channels (pointwise convolution processing). This eliminates the need for multiple image transformations, thereby saving computational resources and improving the speed of feature extraction.

[0188] See also Figure 3H In step 1042, feature mapping is performed based on the initial feature map to obtain candidate feature maps.

[0189] In some embodiments, see Figure 3J , Figure 3J This is a schematic diagram of the tenth process of the visual resource detection method provided in the embodiments of this application. Figure 3H Step 1042 can be achieved through Figure 3J Steps 10421 to 10423 are implemented, and the details are explained below.

[0190] In step 10421, sliding sampling is performed on the initial feature map based on a convolutional kernel of a preset size to obtain multiple local regions, wherein the size of the local regions is the same as that of the convolutional kernel.

[0191] In some embodiments, a predefined convolutional kernel is a fixed-size matrix, which can be an odd number of values, such as 3x3. Each element of the convolutional kernel has an initialized weight value. This convolutional kernel is slid across the input feature map, covering a local region of the initial feature map, and each element of that region is multiplied by the corresponding element of the convolutional kernel.

[0192] For example, if the input feature map is {[1, 2, 3], [4, 5, 6], [7, 8, 9]}, and the weights of the convolution kernel are {[-1, -2, -3], [-4, -5, -6], [-7, -8, -9]}.

[0193] In step 10422, for each local region, the product of the elements in the local region and the corresponding elements in the convolution kernel is determined, and the first summation of the multiple products corresponding to the multiple elements in the local region is determined.

[0194] For example, following the example of step 10421 above, if the local region is [1, 2, 3], then the product is [1*-1, 2*-2, 3*-3] = [-1, -4, -9], and the first sum is -1 + -4 + -9 = -14.

[0195] In step 10423, candidate feature maps are constructed based on multiple first sums corresponding to multiple local regions.

[0196] For example, following the examples of steps 10421 to 10422 above, according to the calculation method of the first summation, the candidate feature map can be obtained as {[-14, -16, -26], [-26, -38, -44], [-49, -54, -17]}.

[0197] See also Figure 3H In step 1043, pooling is performed based on the candidate feature map to obtain the target feature map.

[0198] In some embodiments, see Figure 5 Candidate feature maps can be pooled using either max pooling or average pooling. Max pooling divides the candidate feature map into multiple candidate regions, selecting the maximum value from each region as the output value to retain the most salient features. Average pooling, on the other hand, takes the average value of the values ​​within each candidate region as the output value to obtain an average representation of the overall features.

[0199] For example, following the example in step 10423 above, taking max pooling as an example, for the candidate region [-14, -16, -26], the maximum value is -14; for the candidate region [-26, -38, -44], the maximum value is -26; for the candidate region [-49, -54, -17], the maximum value is -17. Then the target feature map is {[-14], [-26], [-17]}.

[0200] In step 1044, the target feature map is encoded to obtain the image feature vector.

[0201] In some embodiments, see Figure 5The target feature map can be encoded in various ways to obtain the image feature vector, such as one-hot encoding, label encoding, ordinal encoding, and embedding.

[0202] In some embodiments, see Figure 3K , Figure 3K This is a schematic diagram of the eleventh step of the visual resource detection method provided in the embodiments of this application. Figure 3H Step 1044 can be achieved through Figure 3K Steps 10441 to 10445 are implemented, and the details are explained below.

[0203] In step 10441, the target feature map is flattened to obtain a feature map sequence.

[0204] In some embodiments, the flattening operation converts a multidimensional vector into a one-dimensional vector in sequence, that is, the first element of the next row is appended to the last element of the previous row.

[0205] For example, following the example of step 1043 above, after the target feature map {[-14], [-26], [-17]} is flattened, the feature map sequence is obtained as [-14, -26, -17].

[0206] In step 10442, the feature map sequence is embedded to obtain a position embedding vector sequence.

[0207] In some embodiments, one-hot encoding can be used as the embedding method. One-hot encoding is a method of converting categories into binary vectors where only one element is 1 and the rest are 0. Each feature map value in the feature map sequence is treated as a category and assigned a position.

[0208] For example, following the example in step 10441 above, each element in the feature map sequence [-14, -26, -17] is mapped according to its position to obtain a position embedding vector sequence. The first element is mapped to the vector [1, 0, 0], the second element is mapped to the vector [0, 1, 0], and the third element is mapped to the vector [0, 0, 1]. The position embedding vector sequence is {[1, 0, 0][0, 1, 0][0, 0, 1]}.

[0209] In step 10443, the feature map sequence and the location embedding vector sequence are fused into an input vector sequence.

[0210] For example, following the examples from steps 10441 to 10442 above, the input vector sequence is {[-14, 1, 0, 0][-26, 0, 1, 0][-17, 0, 0, 1]}.

[0211] In step 10444, the similarity between any two elements in the input vector sequence is determined.

[0212] In some embodiments, similarity can be calculated using Euclidean distance, Manhattan distance, cosine similarity, Hamming distance, or Jaccard similarity.

[0213] For example, following the example of step 10443 above, taking Euclidean distance as an example, the similarity between any two elements in the input vector sequence is calculated. That is, the sum of the squares of the differences between the corresponding values ​​of the two elements is calculated, and then the square root of the sum of squares is taken as the similarity. The similarity between elements [-14, 1, 0, 0] and [-26, 0, 1, 0] is sqrt((-14-26)^2+(1-0)^2+(0-1)^2+(0-0)^2)=14; the similarity between elements [-14, 1, 0, 0] and [-17, 0, 0, 1] is sqrt((-14-17)^2+(1-0)^2+(0-0)^2+(0-1)^2)=10; and the similarity between elements [-26, 0, 1, 0] and [-17, 0, 0, 1] is sqrt((-26-17)^2+(0-0)^2+(1-0)^2+(0-1)^2)=9.

[0214] In step 10445, multiple elements are weighted and summed based on similarity to obtain the image feature vector.

[0215] For example, following the examples from steps 10441 to 10444 above, the image feature vector is [-14*14, -26*10, -17*9].

[0216] This application embodiment learns representative features from the input image by sequentially performing feature extraction, feature mapping, and pooling, fully reflecting the characteristics of the input image. Furthermore, the extracted features are simplified, reducing computational costs and improving prediction efficiency.

[0217] In some embodiments, feature extraction can be achieved using the encoder of a Transformer, see [link to relevant documentation]. Figure 6 , Figure 6This is a schematic diagram of the Transformer encoder provided in an embodiment of this application. For the input image, the input image is first segmented into fixed-size image patches; then, the segmented image patches are flattened into a sequence, which is a fixed-length vector sequence, where each vector represents an image patch; next, an embedding layer is applied to each element of the sequence to introduce positional information and other contextual information. For image patches, the embedding layer can include positional embedding and linear embedding; finally, the Transformer encoder processes the embedded sequence, using a multi-head attention mechanism to assign weights to each element in the embedded sequence and performing weighted summation to obtain a new sequence representation. Then, the new sequence representation is mapped to a high-dimensional space through fully connected layers and intermediate layers in a feedforward neural network to capture the complex relationships between elements in the sequence, ultimately obtaining the image feature vector.

[0218] See also Figure 3A In step 105, version update detection results of the first visual resource and the second visual resource are obtained based on image feature vectors. The version update detection results indicate whether the update of the first visual resource relative to the second visual resource is normal.

[0219] In some embodiments, when in a cross-version visual resource update scenario, see [link to relevant documentation]. Figure 3L , Figure 3L This is a schematic diagram of the twelfth step of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 105 can be achieved through Figure 3L Steps 1051A to 1053A are implemented, and the details are explained below.

[0220] In some embodiments, taking a game scene as an example, when visual resources are updated across versions, that is, when the game version is significantly updated, the second visual resource should have obvious differences from the first visual resource during the process of updating from the first visual resource to the second visual resource.

[0221] In step 1051A, the first similarity between the first visual resource and the second visual resource is determined based on the image feature vector.

[0222] In some embodiments, the first similarity can be calculated using Euclidean distance, Manhattan distance, cosine similarity, Hamming distance, or Jaccard similarity.

[0223] In step 1052A, in response to the first similarity being less than the similarity threshold, the update of the first visual resource relative to the second visual resource is determined to be a normal version update detection result.

[0224] For example, if the similarity threshold is 0.6 and the first similarity is 0.4, it means that the version update detection result is normal.

[0225] In step 1053A, in response to a first similarity greater than or equal to a similarity threshold, the update of the first visual resource relative to the second visual resource is determined to be an abnormal version update detection result.

[0226] For example, if the similarity threshold is 0.6 and the first similarity is 0.8, it means that the version update detection result is abnormal.

[0227] In some embodiments, when in a scenario of visual resource updates within a version, see [link to relevant documentation]. Figure 3M , Figure 3M This is a schematic diagram of the thirteenth step of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 105 can also be done through Figure 3M Steps 1051B to 1053B are implemented, and the details are explained below.

[0228] In some embodiments, taking a game scene as an example, if the visual resource update within a version represents a minor update to some visual resources within the game, then during the process of updating from the first visual resource to the second visual resource, the second visual resource should not have a significant difference from the first visual resource.

[0229] In step 1051B, a second similarity between the first visual resource and the second visual resource is determined based on the image feature vector.

[0230] In some embodiments, the second similarity can be calculated using Euclidean distance, Manhattan distance, cosine similarity, Hamming distance, or Jaccard similarity.

[0231] In step 1052B, in response to the second similarity being greater than or equal to the similarity threshold, the update of the first visual resource relative to the second visual resource is determined to be a normal version update detection result.

[0232] For example, if the similarity threshold is 0.6 and the second similarity is 0.8, it means that the version update detection result is normal.

[0233] In step 1053B, in response to the second similarity being less than the similarity threshold, the update of the first visual resource relative to the second visual resource is determined to be an abnormal version update detection result.

[0234] For example, if the similarity threshold is 0.6 and the second similarity is 0.4, it indicates that the version update detection result is abnormal.

[0235] This application embodiment enhances the flexibility of visual resource detection and improves the adaptability of visual resource detection methods in different scenarios by setting different detection standards in different scenarios.

[0236] In some embodiments, see Figure 3N , Figure 3N This is a schematic diagram of the fourteenth step of the visual resource detection method provided in the embodiments of this application. Figure 3A Step 105 can also be done through Figure 3N Steps 1051C to 1054C are implemented, and the details are explained below.

[0237] In step 1051C, an initial machine learning model and a training set are obtained. The training set includes multiple training samples. Each training sample includes a sample image feature vector determined based on the first visual resource sample and the second visual resource sample, as well as a real label. The real label represents the version update detection result of the first visual resource sample and the second visual resource sample.

[0238] For example, the machine learning model can be a Transformer model, a Latent Dirichlet Allocation (LDA) model, or a Multilayer Perceptron (MLP). Initializing the representation involves randomly assigning values ​​to the model's parameters, such as assigning all parameters to 0 or all parameters to 1.

[0239] In step 1052C, the machine learning model is initialized based on the feature vector of the sample image to obtain the predicted label.

[0240] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating the principle of training a machine learning model according to an embodiment of this application. Figure 7 In this process, based on time-node samples, the convolutional and fully connected layers of the machine learning model can be initialized to obtain predicted labels.

[0241] In step 1053C, the loss value is determined based on the true label and the predicted label, and the parameters of the initial machine learning model are updated based on the loss value to obtain the trained machine learning model.

[0242] In some embodiments, the difference between the true label and the predicted label is calculated as the loss value. The loss value is backpropagated to update the parameters of the initial machine learning model. This process of calculating the loss value and updating the parameters is repeated multiple times until the loss value no longer increases or decreases. The iteration process is then stopped, and the trained machine learning model is formed.

[0243] For example, the loss function can be the mean squared error loss function, the cross-entropy loss function, the multi-label classification loss function, and the triplet loss function.

[0244] For example, backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer and updates the neuron's weights and biases based on the gradients. Gradient descent is used to continuously update the parameters, thus reducing the loss value. Various gradient descent algorithms can be used, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0245] In step 1054C, the trained machine learning model is invoked based on the image feature vector to obtain the version update detection results of the first visual resource and the second visual resource.

[0246] In some embodiments, if the machine learning model is a Transformer model, the image feature vector is input into the encoder of the trained Transformer model for encoding, and the resulting encoded vector is input into the decoder. Through a linear transformation layer and a Softmax layer, the version update detection results of the first visual resource and the second visual resource are obtained.

[0247] In some embodiments, the version update detection result can be the similarity between the first visual resource and the second visual resource, or it can be a binary classification result, such as similar or dissimilar.

[0248] This application embodiment uses only an encoder, which, compared to the encoder-decoder structure used in the prior art, reduces the length of the encoded token and effectively improves the speed of model training and derivation.

[0249] The following will describe an exemplary application of the embodiments of this application in a visual resource detection scenario during a game version update.

[0250] The embodiments of this application are not limited to the game visual resource detection scenario, but are also applicable to other scenarios, such as the visual resource detection scenario when the software interface is updated or the visual resource detection scenario in media resources.

[0251] Taking game scenes as an example, visual resources refer to the textures and materials of character models, environment backgrounds, virtual props, items, special effects, and lighting effects in the game. When a game version is updated, visual resources in the old and new versions at the same time often exhibit anomalies due to the update. Related technologies typically involve capturing frames of visual resources from different versions at a fixed point in time (i.e., the same moment), and then manually comparing the differences between frames at the same moment to obtain the results. When a game includes a large number of visual resources, the detection methods of these technologies become too costly. Furthermore, new versions may have changes in configuration information or bug fixes, meaning that the movements of models and virtual characters in the game scene may not be aligned with the current frame of a previous version, but rather with the frames before and after it. Using only image frames at the same moment in these technologies can lead to inaccurate detection results. The visual resource detection method provided in this application involves obtaining a first image frame corresponding to a first moment from a first video including a first visual resource; obtaining a second image frame corresponding to the first moment from a second video including a second visual resource, and obtaining at least one third image frame adjacent to the second image frame, wherein the first visual resource is an updated version of the second visual resource; concatenating the first image frame, the second image frame, and at least one third image frame into an input image according to channels; extracting features from the input image to obtain an image feature vector; and obtaining the version update detection result of the first visual resource and the second visual resource based on the image feature vector. This method overlays the third image frame associated with the second image frame during the process of obtaining the version update detection result, thereby mining image frames in historical versions that will affect the version update detection result from multiple perspectives. By utilizing the correlation between multiple adjacent image frames of historical versions and the first image frame of the new version, the accuracy of detection is improved.

[0252] The visual resource detection method provided in this application fuses multiple image channels and frame difference channels, then obtains the final feature map through a residual network, feature mapping, and max pooling. The final feature map is then input into the encoder of a Transformer for feature encoding. Finally, the sequence vector output by the encoder is passed through a multilayer perceptron to obtain the judgment result (i.e., the version update detection result). See [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram illustrating the determination result provided in the embodiments of this application. Figure 8 In the image, the right side shows the new version's image frame (i.e., the first image frame), and the left side shows the old version's image frame (i.e., the second image frame). The status (i.e., the version update detection result) is normal. The implementation process will be described in detail below.

[0253] Data processing section: To effectively utilize the information from the historical frame B1 (the second image frame) corresponding to the current frame A (the first image frame) and the preceding and following frames B2 and B3 (which are at least one third image frame), the RGB channels of the four images A, B1, B2, and B3 are sequentially accumulated, resulting in 3 channels for each image and a total of 4*3=12 channels. Simultaneously, to learn the difference between the new version and historical versions, a frame difference feature layer is superimposed on the input channels. Considering that if the current frame A of the new version contains model loss or model anomalies, the difference with the historical version will inevitably show significant differences compared to the three images in the B series, namely historical frame B1 and its preceding and following frames B2 and B3. In this case, the visual resources should be considered normal. If the missing model in the current frame A is due to time deviation, for example, the action and effects of the historical frame B1 should appear in the current frame A, but only the model corresponds to the historical frame B1, and the effects correspond to the previous historical frame B2, then when calculating the difference separately, the difference between the model part of the current frame A and the historical frame B1 is small, and the difference between the effects part of the current frame A and the previous historical frame B2 is small. In this case, it should be judged as normal. In view of this, the specific implementation of the frame difference feature layer is as follows: (1)

[0254] Diff=min (|A-B1|, |A-B2|, |A-B3|) (1)

[0255] Here, || represents taking the absolute value, min() is used to find the minimum value, and Diff is the frame difference. Let the height of each frame be H and the width be W. Then the input size after data processing is [15, H, W]. Here, 15 represents 15 channels, that is, the input consists of four images A, B1, B2 and B3. All images have 3 channels, for a total of 4*3=12 channels. The frame difference is also an image with 3 channels, so the total is 15 channels.

[0256] Feature Map Creation: Input tokens for the image Transformer are generally obtained through two methods: block partitioning combined with patch masks or using features extracted by convolutional networks. Using features from convolutional networks as token sequences effectively preserves the spatial locality of the image, facilitating the capture of local patterns and features. Compared to directly using block partitioning, this reduces computational load and lowers computational costs. Since the attention given to each channel and different regions within a channel varies during feature extraction, a convolutional attention mechanism is used to learn the contributions of different channels and spatial regions to the generated features. The ResNet50 residual network contains a total of five convolutional layers. A convolutional attention mechanism is added after each convolutional layer except the first one. This is because the first convolutional layer is responsible for learning low-level features, which are important throughout the image. Subsequent convolutional layers need to learn high-level features, which often only appear in certain regions or channels of the image. Therefore, CBAM is used in subsequent layers but not in the first layer. To reduce the computational cost of the Transformer encoder, the input dimension is reduced through feature mapping and max pooling, setting the encoder channel dimension D to 32. The feature map (i.e., the initial feature map) obtained by ResNet50 as the feature extraction network has a size of [W / 32, H / 32, 2048]. In order to adapt to the input channel dimension of the Transformer encoder, convolutional layers are used for feature mapping to map the feature map to a specific size [W / 32, H / 32, 32] (i.e., candidate feature map). At the same time, max pooling is used to reduce the number of tokens. Let the coefficient of max pooling be P, then the size of the final feature map (i.e., target feature map) is [W / 32 / P, H / 32 / P, 32].

[0257] Feature encoding classification: Compared to using the entire Transformer encoder-decoder structure to reconstruct the changing parts of the scene, only the encoder structure of the Transformer is used. The obtained feature map (target feature map) and the learnable position embedding (i.e., position embedding vector sequence) are input into the encoder. The sequence vector (i.e. image feature vector) output by the encoder, which contains the feature vector and the attention distribution of each position, is used to obtain the classification result through a multilayer perceptron.

[0258] See Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of the visual resource detection method provided in this application embodiment. Figure 9 In this context, visual resource detection is achieved by performing the following operations:

[0259] First, preprocess the four images A, B1, B2, and B3 by sequentially accumulating the four images and the frame difference. Each image has 3 channels, so there are a total of 15 channels. The height of each frame image is H, and the width of the image is W. Therefore, the input size after preprocessing is [W, H, 15].

[0260] Secondly, feature extraction is performed using the ResNet50 feature extractor, which includes five convolutional layers. After each convolutional layer (C2, C3, C4, and C5) except for the first convolutional layer (C1), a convolutional attention mechanism (CBAM) module is added. The size of the feature map obtained after feature extraction is [W / 32, H / 32, 2048].

[0261] Then, convolutional layers are used to perform feature mapping on the feature maps obtained after feature extraction, mapping the feature maps to a specific size [W / 32, H / 32, 32]. Next, max pooling is performed on the feature maps of the specific size. Let the coefficient of max pooling be P, then the size of the final feature map is [W / 32 / P, H / 32 / P, 32].

[0262] Finally, the final feature map is flattened and input into the encoder. The sequence vector output by the encoder is passed through a multilayer perceptron to obtain the classification result (i.e., the version update detection result). The classification result includes <the visual resources of the current frame of the new version are normal, and the visual resources of the current frame of the new version are abnormal>.

[0263] This application embodiment obtains an input image by stitching together the first image frame of the new version's visual resources (i.e., the first visual resource) at a first moment, the second image frame of the historical version's visual resources (i.e., the second visual resource) at a first moment, and a third image frame adjacent to the second image frame, according to channels. This allows for the reference of the third image frame associated with the second image frame during the acquisition of version update detection results. It also allows for the mining of image frames from historical versions that might influence the version update detection results from multiple perspectives. By utilizing the correlation between adjacent image frames from historical versions and the first image frame of the new version, the accuracy of detection is improved. By determining the image group to which the second image frame belongs, and selecting the third image frame from the image group, the temporal continuity of the image frames within the same group ensures visual coherence between the selected third image frame and the second image frame. This avoids noise caused by changes in visual resources when image frames are far apart, thus guaranteeing the accuracy of the detection results. By selecting multiple image frames preceding and following the second image frame, and determining the average grayscale value according to each channel, the average grayscale values ​​of each channel are concatenated to form the third image frame. This allows the third image frame to fuse features from multiple adjacent image frames of the second image frame, ensuring that the fusion features of these adjacent image frames are fully learned during the prediction process. This enhances the correlation between the prediction result and the adjacent image frames of the second image frame, improving prediction accuracy. Furthermore, by masking the second and third image frames using a target detection algorithm before concatenating them with the first image frame, only the visual resources in the second and third image frames are retained. This removes noise introduced from areas outside the imaging region in the second and third image frames, allowing the prediction process to focus more on the imaging region and improving prediction accuracy. By stitching together the first image frame, the second image frame, at least one third image frame, and the frame difference into an input image, the differences between image frames can be learned more directly during visual resource detection, improving prediction efficiency. By fusing the frame difference between the first image frame and the second image frame and at least one third image frame into the input image, the influence of image differences is fully considered during prediction, improving prediction accuracy.

[0264] The following description continues to illustrate the exemplary structure of the visual resource detection device 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software module in the visual resource detection device 555 stored in the memory 550 may include:

[0265] The first acquisition module 5551 is used to acquire a first image frame corresponding to a first moment from a first video including a first visual resource.

[0266] The second obtaining module 5552 is used to obtain a second image frame corresponding to a first moment from a second video including a second visual resource, and to obtain at least one third image frame adjacent to the second image frame, wherein the first visual resource is obtained by updating the visual elements in the second visual resource.

[0267] The image stitching module 5553 is used to stitch together a first image frame, a second image frame, and at least one third image frame according to channels to form an input image.

[0268] The feature extraction module 5554 is used to extract features from the input image to obtain the image feature vector.

[0269] The result acquisition module 5555 is used to obtain the version update detection results of the first visual resource and the second visual resource based on the image feature vector. The version update detection results indicate whether the update of the first visual resource relative to the second visual resource is normal.

[0270] In some embodiments, the second obtaining module 5552 is further configured to determine the image group in which the second image frame is located from the second video; and to designate at least one image frame in the image group other than the second image frame as at least one third image frame adjacent to the second image frame.

[0271] In some embodiments, the second obtaining module 5552 is further configured to select at least one forward image frame located before the second image frame; select at least one backward image frame located after the second image frame, wherein the number of at least one forward image frame and at least one backward image frame is the same; traverse at least one forward image frame and at least one backward image frame, determine the average gray value of any currently traversed forward image frame and any backward image frame according to the channel, and stitch the average gray value of each channel together to form a third image frame.

[0272] In some embodiments, the second obtaining module 5552 is further configured to obtain at least one forward image frame from the second video as a third image frame, wherein the timestamp of the forward image frame is located before the timestamp of the second image frame; and to obtain at least one backward image frame from the second video as a third image frame, wherein the timestamp of the backward image frame is located after the timestamp of the second image frame.

[0273] In some embodiments, the number of at least one forward image frame and at least one backward image frame are both preset numbers. The second obtaining module 5552 is further configured to obtain the first frame rate of the first video and the second frame rate of the second video; determine the larger frame rate between the first frame rate and the second frame rate; and use the product of the larger frame rate and a preset coefficient as the preset number.

[0274] In some embodiments, the image stitching module 5553 is further configured to perform target detection on the second image frame and the third image frame to obtain the imaging region of the second visual resource in the second image frame and the third image frame; generate a mask based on the region outside the imaging region; apply the mask to the second image frame and the third image frame respectively to obtain the masked second image frame and the third image frame; and stitch the first image frame, the masked second image frame and the third image frame according to the channel to form an input image.

[0275] In some embodiments, the image stitching module 5553 is further configured to determine a frame difference based on a first image frame, a second image frame, and at least one third image frame; and stitch the first image frame, the second image frame, at least one third image frame, and the frame difference into an input image.

[0276] In some embodiments, the image stitching module 5553 is further configured to: acquire a first grayscale value of a pixel in a first image frame and a second grayscale value of a pixel in a second image frame that is at the same position as a pixel in the first image frame; determine a first difference between the first grayscale value and the second grayscale value; acquire a third grayscale value of a pixel in at least one third image frame that is at the same position as a pixel in the first image frame; determine a second difference between the first grayscale value and the third grayscale value; and determine a minimum difference based on the first difference and the second difference as the frame difference.

[0277] In some embodiments, the feature extraction module 5554 is further configured to perform feature extraction based on the input image to obtain an initial feature map; perform feature mapping based on the initial feature map to obtain a candidate feature map; perform pooling based on the candidate feature map to obtain a target feature map; and perform encoding based on the target feature map to obtain an image feature vector.

[0278] In some embodiments, the feature extraction module 5554 is further configured to perform depthwise convolution processing on the data contained in each channel of the input image to obtain a scalar product corresponding to each channel; stack the scalar products corresponding to each channel; and perform pointwise convolution processing on the stacked scalar products to obtain an initial feature map.

[0279] In some embodiments, the feature extraction module 5554 is further configured to perform sliding sampling on the initial feature map based on a convolutional kernel of a preset size to obtain multiple local regions, wherein the size of the local regions is the same as that of the convolutional kernel; for each local region, determine the product of the elements in the local region with the corresponding elements in the convolutional kernel, determine the first sum of the multiple products corresponding to the multiple elements in the local region respectively; and construct a candidate feature map based on the multiple first sums corresponding to the multiple local regions respectively.

[0280] In some embodiments, the feature extraction module 5554 is further configured to flatten the target feature map to obtain a feature map sequence; perform embedding processing on the feature map sequence to obtain a position embedding vector sequence; fuse the feature map sequence and the position embedding vector sequence into an input vector sequence; determine the similarity between any two elements in the input vector sequence; and perform weighted summation on multiple elements based on the similarity to obtain an image feature vector.

[0281] In some embodiments, the result acquisition module 5555 is further configured to determine a first similarity between the first visual resource and the second visual resource based on the image feature vector; in response to the first similarity being less than a similarity threshold, determine that the update of the first visual resource relative to the second visual resource is a normal version update detection result; and in response to the first similarity being greater than or equal to the similarity threshold, determine that the update of the first visual resource relative to the second visual resource is an abnormal version update detection result.

[0282] In some embodiments, the result acquisition module 5555 is further configured to determine a second similarity between the first visual resource and the second visual resource based on the image feature vector; in response to the second similarity being greater than or equal to a similarity threshold, determine that the update of the first visual resource relative to the second visual resource is a normal version update detection result; and in response to the second similarity being less than a similarity threshold, determine that the update of the first visual resource relative to the second visual resource is an abnormal version update detection result.

[0283] In some embodiments, the result acquisition module 5555 is further configured to acquire an initialized machine learning model and a training set, wherein the training set includes multiple training samples, each training sample including a sample image feature vector determined based on a first visual resource sample and a second visual resource sample, and a ground truth label, the ground truth label representing the version update detection result of the first visual resource sample and the second visual resource sample; the initialized machine learning model is invoked based on the sample image feature vector to obtain a predicted label; a loss value is determined based on the ground truth label and the predicted label, and the parameters of the initialized machine learning model are updated based on the loss value to obtain a trained machine learning model; the trained machine learning model is invoked based on the image feature vector to obtain the version update detection results of the first visual resource and the second visual resource.

[0284] This application provides a computer program product comprising a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the visual resource detection method described above in this application.

[0285] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the visual resource detection method provided in this application. For example, ... Figure 3A The method for detecting visual resources is shown.

[0286] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0287] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0288] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0289] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0290] In summary, by stitching together the first image frame of the new version's visual resources (i.e., the first visual resource) at the first moment, the second image frame of the historical version's visual resources (i.e., the second visual resource) at the first moment, and the third image frame adjacent to the second image frame in the embodiments of this application, an input image is obtained. This allows for the reference of the third image frame associated with the second image frame during the acquisition of version update detection results. Image frames from the historical version that influence the version update detection results are extracted from multiple perspectives. The correlation between adjacent image frames from the historical version and the first image frame of the new version is utilized, improving detection accuracy. By determining the image group to which the second image frame belongs and selecting the third image frame from the image group, the temporal continuity of the image frames within the same image group ensures visual coherence between the selected third image frame and the second image frame. This avoids noise caused by changes in visual resources when image frames are far apart, thus guaranteeing the accuracy of the detection results. By selecting multiple image frames preceding and following the second image frame, and determining the average grayscale value according to each channel, the average grayscale values ​​of each channel are concatenated to form the third image frame. This allows the third image frame to fuse features from multiple adjacent image frames of the second image frame, ensuring that the fusion features of these adjacent image frames are fully learned during the prediction process. This enhances the correlation between the prediction result and the adjacent image frames of the second image frame, improving prediction accuracy. Furthermore, by masking the second and third image frames using a target detection algorithm before concatenating them with the first image frame, only the visual resources in the second and third image frames are retained. This removes noise introduced from areas outside the imaging region in the second and third image frames, allowing the prediction process to focus more on the imaging region and improving prediction accuracy. By stitching together the first image frame, the second image frame, at least one third image frame, and the frame difference into an input image, the differences between image frames can be learned more directly during visual resource detection, improving prediction efficiency. By fusing the frame difference between the first image frame and the second image frame and at least one third image frame into the input image, the influence of image differences is fully considered during prediction, improving prediction accuracy.

[0291] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method of detecting visual resources, characterized in that, The method comprises: obtaining a first image frame corresponding to a first time point from a first video comprising a first visual resource; obtaining a second image frame corresponding to the first time point from a second video comprising a second visual resource, and obtaining at least one third image frame adjacent to the second image frame, wherein the first visual resource is obtained by updating a visual element in the second visual resource; stitching the first image frame, the second image frame and the at least one third image frame into an input image according to channels; performing feature extraction on the input image to obtain an image feature vector; obtaining a version update detection result of the first visual resource and the second visual resource based on the image feature vector, wherein the version update detection result represents whether the update of the first visual resource relative to the second visual resource is normal.

2. The method of claim 1, wherein, The obtaining of the at least one third image frame adjacent to the second image frame comprises: performing at least one of the following processes: obtaining at least one forward image frame from the second video as a third image frame, wherein the timestamp of the forward image frame is located before the timestamp of the second image frame; obtaining at least one backward image frame from the second video as a third image frame, wherein the timestamp of the backward image frame is located after the timestamp of the second image frame.

3. The method of claim 2, wherein, The number of the at least one forward image frame and the at least one backward image frame is a preset number. Before the obtaining of the at least one third image frame adjacent to the second image frame, the method further comprises: obtaining a first frame rate of the first video and a second frame rate of the second video; determining a larger frame rate from the first frame rate and the second frame rate; based on the product of the larger frame rate and a preset coefficient as the preset number.

4. The method of any one of claims 1 to 3, wherein The obtaining of the at least one third image frame adjacent to the second image frame comprises: determining a group of images in which the second image frame is located from the second video; taking at least one image frame in the group of images other than the second image frame as at least one third image frame adjacent to the second image frame.

5. The method according to any one of claims 1 to 4, characterized in that, The stitching of the first image frame, the second image frame and the at least one third image frame into an input image according to channels comprises: performing target detection on the second image frame and the third image frame to obtain an imaging area of the second visual resource in the second image frame and the third image frame; generating a mask according to an area other than the imaging area; applying the mask to the second image frame and the third image frame respectively to obtain the second image frame and the third image frame after the mask is applied; stitching the first image frame, the second image frame and the third image frame after the mask is applied respectively into an input image according to channels.

6. The method according to any one of claims 1 to 3, characterized in that, The obtaining of the at least one third image frame adjacent to the second image frame comprises: selecting at least one forward image frame located before the second image frame; select at least one backward image frame located after the second image frame, wherein the number of the at least one forward image frame is the same as that of the at least one backward image frame; traverse the at least one forward image frame and the at least one backward image frame, and splice average gray values of each channel into a third image frame according to the channel, wherein the average gray values are determined by any one of the forward image frame and any one of the backward image frame being currently traversed.

7. The method according to any one of claims 1 to 6, characterized in that, When being in a cross-version visual resource update scene, the obtaining of the version update detection result of the first visual resource and the second visual resource based on the image feature vector comprises: determining a first similarity of the first visual resource and the second visual resource based on the image feature vector; in response to the first similarity being less than a similarity threshold, determining that the update of the first visual resource relative to the second visual resource is a normal version update detection result; in response to the first similarity being greater than or equal to the similarity threshold, determining that the update of the first visual resource relative to the second visual resource is an abnormal version update detection result.

8. The method of any one of claims 1 to 6, wherein When being in an intra-version visual resource update scene, the obtaining of the version update detection result of the first visual resource and the second visual resource based on the image feature vector comprises: determining a second similarity of the first visual resource and the second visual resource based on the image feature vector; in response to the second similarity being greater than or equal to a similarity threshold, determining that the update of the first visual resource relative to the second visual resource is a normal version update detection result; in response to the second similarity being less than the similarity threshold, determining that the update of the first visual resource relative to the second visual resource is an abnormal version update detection result.

9. The method according to any one of claims 1 to 8, characterized in that, The splicing of the first image frame, the second image frame and the at least one third image frame into an input image comprises: determining a frame difference based on the first image frame, the second image frame and the at least one third image frame; splicing the first image frame, the second image frame, the at least one third image frame and the frame difference into an input image.

10. The method of claim 9, wherein, The determining of the frame difference based on the first image frame, the second image frame and the at least one third image frame comprises: obtaining a first gray value of a pixel point of the first image frame and a second gray value of a pixel point in the same position as the pixel point of the first image frame in the second image frame; determining a first difference value of the first gray value and the second gray value; obtaining a third gray value of a pixel point in the same position as the pixel point of the first image frame in the at least one third image frame; determining a second difference value of the first gray value and the third gray value; determining a minimum difference value based on the first difference value and the second difference value as a frame difference.

11. The method according to any one of claims 1 to 10, characterized in that, The feature extraction on the input image to obtain an image feature vector comprises: performing feature extraction based on the input image to obtain an initial feature map; performing feature mapping based on the initial feature map to obtain a candidate feature map; pooling based on the candidate feature map to obtain a target feature map; encoding based on the target feature map to obtain an image feature vector.

12. The method of claim 11, wherein, The feature extraction based on the input image obtains an initial feature map, including: performing depth-wise convolution processing on data contained in each channel of the input image to obtain scalar products corresponding to each channel; stacking the scalar products corresponding to each channel and performing point-wise convolution processing on the stacked scalar products to obtain an initial feature map.

13. The method of claim 11, wherein, The feature mapping based on the initial feature map obtains a candidate feature map, including: sliding sampling in the initial feature map based on a preset size of a convolution kernel to obtain a plurality of local regions, wherein the local regions are the same size as the convolution kernel; for each local region, determining the product of the elements in the local region and the elements at the corresponding positions in the convolution kernel, determining the first sum of a plurality of products corresponding to a plurality of elements in the local region, respectively; constructing a candidate feature map based on a plurality of first sums corresponding to a plurality of local regions, respectively.

14. The method of claim 11, wherein, The encoding based on the target feature map obtains an image feature vector, including: performing flattening operation on the target feature map to obtain a feature map sequence; performing embedding processing on the feature map sequence to obtain a position embedding vector sequence; fusing the feature map sequence and the position embedding vector sequence into an input vector sequence; determining the similarity between any two elements in the input vector sequence; performing weighted summation on a plurality of elements based on the similarity to obtain an image feature vector.

15. The method according to any one of claims 1 to 14, characterized in that, The version update detection result of the first visual resource and the second visual resource based on the image feature vector includes: obtaining an initialized machine learning model and a training set, wherein the training set includes a plurality of training samples, each training sample includes a sample image feature vector determined based on a first visual resource sample and a second visual resource sample and a real label, and the real label represents the version update detection result of the first visual resource sample and the second visual resource sample; calling the initialized machine learning model based on the sample image feature vector to obtain a predicted label; determining a loss value based on the real label and the predicted label, and updating the parameters of the initialized machine learning model based on the loss value to obtain a trained machine learning model; calling the trained machine learning model based on the image feature vector to obtain the version update detection result of the first visual resource and the second visual resource.

16. A device for detecting visual resources, characterized in that The device includes: a first obtaining module for obtaining a first image frame corresponding to a first time from a first video including a first visual resource; a second obtaining module for obtaining a second image frame corresponding to the first time from a second video including a second visual resource, and obtaining at least one third image frame adjacent to the second image frame, wherein the first visual resource is obtained by updating a visual element in the second visual resource; The image splicing module is configured to splice the first image frame, the second image frame and the at least one third image frame into an input image according to a channel; The feature extraction module is configured to perform feature extraction on the input image to obtain an image feature vector; The result obtaining module is configured to obtain a version update detection result of the first visual resource and the second visual resource based on the image feature vector, wherein the version update detection result represents whether the update of the first visual resource relative to the second visual resource is normal.

17. An electronic device, comprising: The electronic device comprises: A memory configured to store computer executable instructions; A processor configured to execute the computer executable instructions stored in the memory to implement the visual resource detection method in any one of claims 1 to 15.

18. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program comprise the steps of claim 17. The computer executable instructions or the computer program are executed by the processor to implement the visual resource detection method in any one of claims 1 to 15.

19. A computer program product comprising computer-executable instructions or a computer program, characterized in that, The computer executable instructions or the computer program are executed by the processor to implement the visual resource detection method in any one of claims 1 to 15.

Images