Image processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an image processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a first image and a second image, performing block segmentation processing based on target identification on the first image to obtain at least one first block of the first image, and performing matching processing on each first block and the second image to obtain at least one second block of the first image; obtaining a first matching block matched with the first block in the second image and a first matching degree between the first block and the first matching block, and determining a first difference degree between each first block and the corresponding first matching block, and based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, determining a first block, which is different from the content of the second image, in the first image. Through the method and the device, the accuracy of positioning the difference block can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an image processing method and device, an electronic device, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which enables machines to have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] There are many application scenarios for block comparison in images. Taking a software development scenario as an example, in the software development process, in order to avoid the problem that the actual display interface does not match the theoretical display interface, the related technology directly compares the two interfaces as a whole, resulting in poor comparison effect. SUMMARY

[0004] The embodiments of the present application provide an image processing method and device, an electronic device, a computer readable storage medium, and a computer program product, which can improve the accuracy of locating difference blocks.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide an image processing method, which comprises the following steps:

[0007] obtaining a first image and a second image;

[0008] performing target recognition-based block segmentation processing on the first image to obtain at least one first block of the first image;

[0009] for each first block, performing matching processing on the first block and the second image to obtain a first matching block in the second image that matches the first block, and a first matching degree between the first block and the first matching block;

[0010] determining a first difference degree between each first block and the corresponding first matching block;

[0011] based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, determining a first block in the first image that has a difference in content from the second image.

[0012] The embodiment of the present application provides an image processing device, comprising:

[0013] a data acquisition module, configured to acquire a first image and a second image;

[0014] a block segmentation module, configured to perform target recognition-based block segmentation processing on the first image to obtain at least one first block of the first image;

[0015] an image matching module, configured to, for each first block, perform matching processing on the first block and the second image to obtain a first matching block matched with the first block in the second image and a first matching degree between the first block and the first matching block;

[0016] a difference degree determination module, configured to determine a first difference degree between each first block and a corresponding first matching block;

[0017] a block determination module, configured to determine, based on the first matching degree and the first difference degree between each first block and a corresponding first matching block, a first block in the first image that has a difference in content with the second image.

[0018] In the above scheme, the block segmentation module is further configured to perform target recognition processing on the first image to obtain a detection frame corresponding to each target recognition result; and perform segmentation processing on the first image based on at least one detection frame to obtain a first block corresponding to each detection frame.

[0019] In the above scheme, the image matching module is further configured to perform intercepting processing on the second image based on the first block to obtain a first to-be-matched block corresponding to each first block in the second image; and perform matching processing on the first block and the corresponding first to-be-matched block to obtain a first matching block matched with the first block in the first to-be-matched block and a first matching degree between the first block and the first matching block.

[0020] In the above scheme, the image matching module is further configured to determine a positioning coordinate of the first block in the first image; determine a query positioning coordinate based on the positioning coordinate of the first block in the first image; and perform intercepting processing on the second image based on the query positioning coordinate to obtain a first to-be-matched block corresponding to the first block in the second image.

[0021] In the scheme, the image matching module is further configured to: perform scaling processing on any of the first image and the second image to obtain a scaled first image or a scaled second image, wherein the scaled image has the same size as the unscaled image; obtain a size parameter difference between the image before the scaling processing and the image after the scaling processing for the scaled image; and adjust the positioning coordinates of the first block in the first image based on the size parameter difference to obtain the query positioning coordinates.

[0022] In the scheme, the image matching module is further configured to: take the size of the first block as a search box size, and perform traversal processing on the first block to be matched using the search box to obtain at least one traversal block; when the number of the traversal blocks is one, take the traversal block as a first matching block matched with the first block, and compare the first matching block with the first block to obtain a first matching degree of the first matching block and the first block; and when the number of the traversal blocks is more than one, compare each of the traversal blocks with the first block to obtain a first matching degree of each of the traversal blocks and the first block, and take the traversal block with the highest first matching degree as the first matching block matched with the first block.

[0023] In the scheme, the image matching module is further configured to: obtain a first pixel of each of the pixel positions in the first block; and perform the following processing for each of the traversal blocks: obtain a second pixel of each of the pixel positions in the traversal block; determine, for each of the pixel positions, a pixel distance between the first pixel of the pixel position and the second pixel of the pixel position; perform fusion processing on the pixel distances of the plurality of pixel positions to obtain an overall pixel distance of the traversal block and the first block; and obtain a first matching degree negatively correlated with the overall pixel distance.

[0024] In the scheme, the difference degree determination module is further configured to: perform the following processing for each of the first blocks: determine a first histogram of the first block and a second histogram of the first matching block corresponding to the first block; determine a correlation degree between the first histogram and the second histogram; and obtain a first difference degree negatively correlated with the correlation degree.

[0025] In the scheme, the block determination module is further configured to: for each first block, acquire a difference score corresponding to the first block, the difference score being positively related to a first matching degree corresponding to the first block and negatively related to a first difference degree corresponding to the first block; and when the difference score is less than a difference score threshold, determine the first block as a first block in which the first image has a difference with the second image.

[0026] In the scheme, the block determination module is further configured to: perform target recognition-based block segmentation on the second image to obtain at least one second block of the second image; for each second block, perform matching between the second block and the first image to obtain a second matching block in the first image that matches the second block and a second matching degree between the second block and the second matching block; determine a second difference degree between each second block and a corresponding second matching block; and based on the second matching degree and the second difference degree between each second block and a corresponding second matching block, determine a second block in the second image that has a difference with the first image.

[0027] In the scheme, the block determination module is further configured to: when at least one of the following conditions is met, perform matching between the second block and the first image to obtain a second matching block in the first image that matches the second block and a second matching degree between the second block and the second matching block: the number of the first blocks is not greater than the number of the second blocks; and the number of the first blocks in which the first image has a difference with the second image is less than a first number threshold.

[0028] An electronic device is provided in the embodiments of the present application, and the electronic device comprises:

[0029] A memory is configured to store computer-executable instructions.

[0030] A processor is configured to execute the computer-executable instructions stored in the memory to implement the image processing method provided in the embodiments of the present application.

[0031] A computer-readable storage medium is provided in the embodiments of the present application, and the computer-readable storage medium stores computer-executable instructions, and is configured to be executed by a processor to implement the image processing method provided in the embodiments of the present application.

[0032] A computer program product is provided in the embodiments of the present application, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the image processing method provided in the embodiments of the present application.

[0033] The embodiments of the present application have the following beneficial effects:

[0034] The first image and the second image are acquired, and a target recognition-based block segmentation processing is performed on the first image to obtain at least one first block of the first image. Since the block segmentation is performed based on target recognition, the obtained first block is a valuable block segmented based on target recognition. Compared with direct segmentation into fixed-size blocks, the calculation burden of subsequent matching processing can be effectively reduced, and the efficiency of subsequent matching processing is improved. Then, for each first block, the first block is matched with the second image to obtain a first matching block matched with the first block in the second image and a first matching degree between the first block and the first matching block. Through the matching processing, the first matching block corresponding to each first block can be obtained, so that the coverage of image detection is improved. Then, a first difference degree between each first block and the corresponding first matching block can be determined, and based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, a first block with a difference in content from the second image in the first image is determined. Through the screening in two directions of the first difference degree and the first matching degree, the recognition ability of the block with a difference is improved, and the accuracy of locating the difference block is improved. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is an architecture schematic diagram of an image processing system 100 provided by an embodiment of the present application;

[0036] Figure 2 is a structure schematic diagram of an electronic device 500 provided by an embodiment of the present application;

[0037] Figure 3A is a first flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0038] Figure 3B is a second flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0039] Figure 3C is a third flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0040] Figure 3D is a fourth flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0041] Figure 4 is a schematic diagram of a first image and a second image provided by an embodiment of the present application;

[0042] Figure 5 is a schematic diagram of a first block provided by an embodiment of the present application;

[0043] Figure 6 is a schematic diagram of a first block with a difference in the first image provided by an embodiment of the present application;

[0044] Figure 7 The flowchart in the actual application scenario is provided for the embodiments of the present application.

[0045] It should be noted that the "first", "second" above are only used to distinguish different schemes, and do not represent the degree of superiority or priority in the implementation process. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be described in further detail below in combination with the drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those skilled in the art without making creative labor are within the scope of protection of the present application.

[0047] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subset of all possible embodiments, and can be combined with each other without conflict.

[0048] In the following description, the terms "first, second, third" are only used to distinguish similar objects, and do not represent the specific order of the objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0049] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuit or memory) or combination thereof. Similarly, one processor (or multiple processors or memory) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0050] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meaning as commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0051] In the embodiments of the present application, the relevant data collection process in the actual application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0052] Before the embodiments of the present application are further described, the terms and names involved in the embodiments of the present application are explained, and the terms and names involved in the embodiments of the present application are applicable to the following explanations.

[0053] 1) Automated testing: generally refers to the automation of software testing, which is to run a system or application under predetermined conditions and evaluate the running results. The preconditions should include normal conditions and abnormal conditions.

[0054] 2) Image segmentation: image segmentation is the process of dividing an image into multiple parts or regions, usually based on the attributes or characteristics of pixels. It can be used to identify different objects or regions in an image. The basic idea of the segmentation method based on neural network is to obtain a linear decision function by training a multilayer perceptron, and then classify the pixels using the decision function to achieve the purpose of segmentation.

[0055] 3) Template matching (matchTemplate): template matching is a technique for finding the most matching (similar) part in one image with another template image. The template is a small known image, and template matching is to search for the target image in a large image. The template is a known target image to be found in the image, and the target image and the template have the same size, direction and image. Through the preset algorithm, the target can be found in the image and its coordinate position can be determined.

[0056] 4) Histogram: an image histogram is a statistical table reflecting the distribution of pixels in an image. The horizontal coordinate represents the type of image pixel, which can be grayscale or color. The vertical coordinate represents the total number of each color value in the image or the percentage of the total number of pixels. An image is composed of pixels, and a histogram reflecting the distribution of pixels can often be used as an important feature of an image. In practical engineering, image histograms have good applications in feature extraction and image matching.

[0057] In the process of software development, the developed program often needs to be tested, and in the process of automated testing, the interfaces of the programs before and after development need to be compared to find the difference area. In the prior art, the obtained complete images of the two program interfaces are directly compared, which can obtain the difference blocks of the two images, but the program interface may be displayed by electronic devices with different screen sizes, resulting in different sizes of the obtained program interface images. At this time, directly comparing the two images may not be able to obtain accurate difference blocks due to the different positions of the controls in the program interface.

[0058] The image processing method and device, the electronic device, the computer readable storage medium, and the computer program product provided in the embodiments of the present application can improve the accuracy of locating the difference block. The image processing method provided in the embodiments of the present application is implemented based on computer vision technology in artificial intelligence technology.

[0059] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include, for example, sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-training model technology, operation / interaction system, mechatronics, etc. Among them, the pre-training model is also called large model or basic model, which can be widely applied to downstream tasks in various directions of artificial intelligence after fine tuning. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0060] Computer vision technology (CV) is a science that studies how to make machines see, and further, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further, it refers to making computer processing a more suitable image for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, and tries to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology brings important changes to the development of computer vision technology. The pre-training model in the field of vision can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, etc.

[0061] The following describes an exemplary application of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as a notebook computer, a tablet computer, a desktop computer, a set-top box, a mobile device (for example, a mobile phone, a portable music player, a personal digital assistant, a dedicated message device, a portable game device), a smart device (for example, a smart phone, a smart speaker, a smart watch, a smart television, a smart home appliance, a smart voice interaction device), a vehicle-mounted terminal, an aircraft, etc. Various types of object terminals, and can also be implemented as a server. In the following, an exemplary application when the device is implemented as a server will be described.

[0062] Referring to Figure 1 , Figure 1Fig. 1 is a schematic diagram of an architecture of an image processing system 100 provided by an embodiment of the present application. To implement a support for an image processing application, a terminal 400 connects to a server 200 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of both.

[0063] The terminal 400 is configured to acquire a first image and a second image corresponding to two programs respectively after running the two programs, and then transmit the first image and the second image to the server 200 through the network 300.

[0064] The server 200 is configured to perform target recognition based block segmentation processing on the first image after receiving the first image and the second image sent by the terminal 400, to obtain at least one first block of the first image. For each first block, the server 200 performs matching processing on the first block and the second image, to obtain a first matching block in the second image that matches the first block, and a first matching degree between the first block and the first matching block. The server 200 determines a first difference degree between each first block and the corresponding first matching block, and determines a first block in the first image that has a difference in content with the second image based on the first matching degree and the first difference degree between each first block and the corresponding first matching block. The server 200 can then return the first block in the first image that has a difference to the terminal 400 through the network, and the terminal 400 can display the first block that has a difference to a user.

[0065] The repair network training method provided by the embodiment of the present application can be applied to an automatic testing scenario of game software and an automatic testing scenario of shopping software. In the automatic testing scenario of game software, a developed game interface can be acquired as a first image, and a game interface in a running process can be acquired as a second image. Then, target recognition based block segmentation processing is performed on the first image to obtain at least one first block of the first image. For each first block, matching processing is performed on the first block and the second image to obtain a first matching block in the second image that matches the first block, and a first matching degree between the first block and the first matching block. A first difference degree between each first block and the corresponding first matching block is determined, and a first block in the first image that has a difference in content with the second image is determined based on the first matching degree and the first difference degree between each first block and the corresponding first matching block.

[0066] In the automated test scene of the shopping software, the interface of the developed shopping software can be acquired as a first image, and the interface of the running shopping software can be acquired as a second image. Then, the first image is subjected to block segmentation processing based on target recognition, to obtain at least one first block of the first image. For each first block, the first block and the second image are subjected to matching processing, to obtain a first matching block in the second image matched with the first block, and a first matching degree between the first block and the first matching block. The first difference degree between each first block and the corresponding first matching block is determined, and based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, a first block in the first image that has a difference in content from the second image is determined.

[0067] In some embodiments, the server 200 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and basic cloud computing services such as big data and artificial intelligence platforms. The terminal 400 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, and the like, but is not limited thereto. The terminal and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application.

[0068] Referring to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device 500 provided by the embodiments of the present application, Figure 2 The electronic device 500 shown in the figure includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 540 in the figure. Figure 2

[0069] The processor 510 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0070] ​The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0071] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices remotely located from the processor 510.

[0072] The memory 550 includes volatile memory or nonvolatile memory, and can also include both volatile and nonvolatile memory. Nonvolatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 550 described in embodiments of the present application is intended to include any suitable type of memory.

[0073] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are illustratively described below.

[0074] The operating system 551 includes systems programs for handling various basic system services and for performing hardware dependent tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services as well as handling hardware dependent tasks;

[0075] The network communication module 552 is for communicating to other electronic devices via one or more (wired or wireless) network interfaces 520, example network interfaces 520 include Bluetooth, wireless fidelity (WiFi), universal serial bus (USB), and the like;

[0076] The presentation module 553 is for enabling presentation of information via one or more output devices 531 associated with the user interface 530 (e.g., display screens, speakers, and the like) (e.g., user interfaces for operating peripheral devices and displaying content and information);

[0077] The input processing module 554 is for detecting and interpreting one or more user inputs or interactions from one or more input devices 532.

[0078] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in software, Figure 2 An image processing apparatus 555 stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: a data acquisition module 5551, a block segmentation module 5552, an image matching module 5553, a difference degree determination module 5554, and a block determination module 5555. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0079] In some other embodiments, the apparatus provided by the embodiments of the present application can be implemented in hardware. As an example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the image processing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs), or other electronic elements.

[0080] Next, the repair network training method provided by the embodiments of the present application is described. As mentioned before, the electronic device implementing the repair network training method provided by the embodiments of the present application can be a terminal, a server, or a combination of both. Therefore, the execution subject of each step will not be repeated in the following description.

[0081] Referring to Figure 3A , Figure 3A is the first flowchart of the image processing method provided by the embodiments of the present application. The steps 101 to 105 shown will be described in combination with Figure 3A

[0082] In step 101, a first image and a second image are acquired.

[0083] As an example, the first image and the second image are images used for subsequent detection. Therefore, the first image and the second image can be images of the same interface taken at different times, or images of the same interface taken on different devices. As shown in the first image and the second image, Figure 4 Figure 4 ​​is a schematic diagram of the first image and the second image provided by an embodiment of the present application. In Figure 4 In the above embodiment, 401 can be the first image, and 402 can be the second image. In Figure 4 In the above embodiment, the size of the first image is greater than that of the second image.

[0084] As an example, in the automatic testing scenario of game software, the program interface of the game in the entire development process can be recorded by an electronic device, and the program image can be intercepted from the recorded program interface after the development is completed, and then the intercepted program image and the program interface after the development are completed can be taken as the first image and the second image, respectively.

[0085] In step 102, the first image is subjected to target recognition-based block segmentation processing to obtain at least one first block of the first image.

[0086] As an example, since meaningless areas often exist in the program interface, in order to reduce the resource consumption of subsequent image matching, the blocks required for subsequent image matching can be identified from the first image, and then the identified blocks are segmented from the first image to obtain the first blocks.

[0087] The process of obtaining the first blocks by image segmentation of the first image will be described in detail below.

[0088] In some embodiments, the target recognition-based block segmentation processing of the first image in step 102 to obtain at least one first block of the first image can be implemented by the following technical solution: performing target recognition processing on the first image to obtain a detection box corresponding to each target recognition result, performing segmentation processing on the first image based on at least one detection box to obtain a first block corresponding to each detection box.

[0089] As an example, the first image can be subjected to target recognition processing first to obtain the detection box included in the first image, and then the detection box is segmented from the first image as the first block.

[0090] As an example, the manner of performing target recognition processing on the first image can be to perform target recognition processing on the first image by a pre-trained target recognition model, so as to obtain the detection box of each target object in the first image. The training method of the specific target recognition model is not specifically limited here, and can be selected according to the actual situation.

[0091] As an example, the target object for target recognition on the first image can be selected according to the application scenario. For example, if the first image is an image of a game interface, the target object for target recognition can be a skill bar of a virtual character included in the game interface, a function control of a backpack, a function control of a mall, a function control of virtual character information, etc. If the first image is an image of a shopping interface, the target object for target recognition can be a commodity name of an item, a display icon of an item, a detailed introduction of an item, a price of an item, a selectable coupon, and a payment control, etc.

[0092] As an example, after target recognition is performed on the first image to obtain a detection box included in the first image, the first image with the detection box can be subjected to image segmentation by an image segmentation model to obtain at least one first block. The training manner of the specific image segmentation model can be selected according to actual conditions, which is not limited herein.

[0093] As an example, in the embodiments of the present application, the block segmentation processing based on target recognition on the first image can be implemented by a model having both target recognition and image segmentation functions, or can be implemented by a target recognition model and an image segmentation model respectively.

[0094] As an example, the block segmentation processing based on target recognition can be implemented by a deep learning model (You Only Look Once 8, YOLO8). YOLO8 is a deep learning model mainly used for real-time target detection. YOLO8 is a single-stage target detection algorithm, and its basic idea is to simultaneously perform target detection and object classification, which can process video streams in real time. YOLO8 is an algorithm implemented in deep learning, which uses a convolutional neural network (CNN) to train input images to recognize and locate objects (such as people, vehicles, etc.) that may be contained in the images. It can realize real-time target detection on mobile devices, and has been widely applied in various practical applications such as autonomous driving, intelligent agriculture, etc.

[0095] As an example, the block segmentation processing based on target recognition can be implemented by a symbolic mathematics system (TensorFlow). TensorFlow provides rich APIs and tools, enabling users to build, train, and deploy various types of machine learning models, including neural networks, convolutional neural networks (CNNs), recurrent neural networks (RNNs), reinforcement learning, etc. The architecture of TensorFlow is very flexible and can run on various hardware platforms.

[0096] As an example, reference can be made to Figure 5 the first block included in the first image, Figure 5 is a schematic view of the first block provided by the embodiments of the present application. In Figure 5Each detection frame corresponds to a first block, as shown in 501 of FIG. 4. Figure 5 The "immediate payment" shown in 501 is one of the first blocks.

[0097] As an example, in the process of obtaining a plurality of first blocks as shown in Figure 5 , first, the target recognition result can be obtained by performing artificial intelligence-based target recognition on the first image 401 in Figure 4 , and then image segmentation is performed on the detection frames included in the target recognition result to obtain the first blocks.

[0098] As an example, the function of target recognition is implemented by a trained target recognition model, and the trained target recognition model has the ability to recognize different function controls, function panels, and function identifiers in the image.

[0099] As an example, in the process of target recognition, the function controls included in the first image can be recognized, such as the "immediate payment" function control shown in 501 of FIG. 4, the function identifiers shown in 502 of FIG. 4, and the text content included in the first image, such as "XX coin resource gift" shown in 503 of FIG. 4. It should be noted that for the text content included in the first image, the individual characters of the text content can also be recognized, such as "coin" shown in 504 of FIG. 4. For the same character being divided into two detection frames (such as "coin" shown in 504), in the subsequent image segmentation process, the first block corresponding to "XX coin resource gift" shown in 503 can be segmented from the first image, and the first block corresponding to "coin" shown in 504 can be segmented from the first image. Figure 5 Figure 5 Figure 5 Figure 5

[0100] Through the above scheme, the first blocks that need to be image matched can be segmented from the first image, reducing the resource consumption in the subsequent image matching process.

[0101] In step 103, for each first block, the first block is matched with the second image to obtain a first matching block in the second image that matches the first block, and a first matching degree between the first block and the first matching block.

[0102] In some embodiments, the matching of the first block with the second image in step 103 to obtain the matching block in the second image that matches the first block, and the first matching degree between the first block and the matching block can be implemented by steps 1031 to 1032 as shown in Figure 3B .

[0103] ​​​​In step 1031, the second image is intercepted based on the first block to obtain a first to-be-matched block in the second image corresponding to each first block.

[0104] As an example, after the first block is determined, the first to-be-matched block can be intercepted from the second image. The detailed process of intercepting the first to-be-matched block from the second image is described below.

[0105] In some embodiments, the interception of the second image based on the first block in step 1031 to obtain the first to-be-matched block in the second image corresponding to each first block can be implemented by steps 10311 to 10313 as shown. Figure 3C

[0106] In step 10311, the positioning coordinates of the first block in the first image are determined.

[0107] As an example, the positioning coordinates of the first block in the first image can be determined first. Specifically, the coordinates of the top-left corner vertex of the first block and the coordinates of the bottom-right corner vertex of the first block can be taken as the positioning coordinates of the first block in the first image.

[0108] As an example, the coordinates of the top-left corner vertex B of the first block A in the first image are (3, 4) and the coordinates of the bottom-right corner vertex C are (6, 7) (the positive half of the Y-axis of the coordinate axis is downward and the positive half of the X-axis is to the right), and the positioning coordinates of the first block A in the first image can be (B3, B4, C6, C7).

[0109] In step 10312, the query positioning coordinates are determined based on the positioning coordinates of the first block in the first image.

[0110] In some embodiments, the determination of the query positioning coordinates based on the positioning coordinates of the first block in the first image in step 10312 can be implemented by the following technical solution: scaling any one of the first image and the second image to obtain a scaled first image or a scaled second image, wherein the scaled image has the same size as the unscaled image; obtaining a size parameter difference between the image before scaling and the image after scaling for the scaled image; and adjusting the positioning coordinates of the first block in the first image based on the size parameter difference to obtain the query positioning coordinates.

[0111] As an example, since the sizes of the first image and the second image can be different, any one of the first image and the second image needs to be scaled before the query positioning coordinates are determined, so that the size of the scaled first image or the scaled second image is the same. ​

[0112] As an example, the first image A has a length of 5 cm and a width of 3 cm. The second image B has a length of 8 cm and a width of 6 cm. In this case, the second image B is scaled, and the width of the second image B is reduced to 3 cm without changing the aspect ratio, and the height of the second image B is 4 cm.

[0113] The above is an example of scaling the second image B to the same width as the first image A. Another scaling method is described below.

[0114] As an example, the first image A has a length of 5 cm and a width of 3 cm. The second image B has a length of 8 cm and a width of 6 cm. In this case, the first image A is scaled, and the width of the first image A is enlarged to 6 cm without changing the aspect ratio, and the length of the first image A is 10 cm.

[0115] The above is an example of scaling the first image or the second image with the same scaled width as a constraint. An example of scaling the first image or the second image with the same scaled length as a constraint is described below.

[0116] As an example, the first image A has a length of 5 cm and a width of 3 cm. The second image B has a length of 10 cm and a width of 8 cm. In this case, the second image B is scaled, and the length of the second image B is reduced to 5 cm without changing the aspect ratio, and the width of the second image B is 4 cm.

[0117] The above is an example of scaling the second image B to the same width as the first image A. Another scaling method is described below.

[0118] As an example, the first image A has a length of 5 cm and a width of 3 cm. The second image B has a length of 10 cm and a width of 8 cm. In this case, the first image A is scaled, and the length of the first image A is enlarged to 10 cm without changing the aspect ratio, and the width of the first image A is 6 cm.

[0119] As an example, after scaling, the difference in size parameters between the scaled image and the image without scaling is determined, and the positioning coordinates of the first image are adjusted according to the difference in size parameters to obtain the query positioning coordinates.

[0120] As an example, the first image A has a length of 5 cm and a width of 3 cm. The second image B has a length of 10 cm and a width of 8 cm. In this case, the first image A is enlarged, and the length of the first image A is enlarged to 10 cm while the aspect ratio is unchanged. In this case, the length of the first image A after scaling is 10 cm, and the width of the first image A is 6 cm. The length of the second image B is 10 cm, and the width of the second image B is 8 cm. Since the length of the first image A is the same as the length of the second image B, and the width of the first image A is different from the width of the second image B, the difference between the width of the first image A and the width of the second image B is obtained, and the size parameter difference is 2 cm.

[0121] As an example, if the size parameter difference is 2 cm, the coordinates of the top-left vertex B of the first image A are (3, 4), and the coordinates of the bottom-right vertex C of the first image A are (6, 7). (The positive half of the Y-axis of the coordinate axis is downward, and the positive half of the X-axis is to the right. Each coordinate unit is 1 cm.) The positioning coordinates of the first image A in the first image can be (B3, B4, C6, C7). In this case, the positioning coordinates can be adjusted to obtain the query positioning coordinates (B1, B2, C8, C9), that is, the horizontal and vertical coordinates of the vertex B are subtracted by the size parameter difference, and the horizontal and vertical coordinates of the vertex C are added by the size parameter difference. The area range surrounded by the query positioning coordinates is larger than the area range surrounded by the positioning coordinates.

[0122] As an example, since the size parameter difference is used to ensure that the area range surrounded by the query positioning coordinates includes the entire area range surrounded by the positioning coordinates, when the size parameter difference is less than the difference threshold, the size parameter difference can be adjusted to the size of the difference threshold.

[0123] As an example, the difference threshold is 3 cm, and the size parameter difference is 1 cm. Since the size parameter difference is less than the difference threshold, the size parameter difference can be adjusted to 3 cm.

[0124] Through the above scheme, the difference block of the first image and the second image with different sizes can be determined, and the generality of the image processing method provided by the embodiments of the present application is improved.

[0125] In step 10313, the second image is intercepted based on the query positioning coordinates to obtain a first to-be-matched block corresponding to the first block in the second image.

[0126] As an example, after obtaining the query positioning coordinates, the range surrounded by the query positioning coordinates can be intercepted from the second image as the first to-be-matched block corresponding to the first block.

[0127] As an example, if the query location coordinates are (B1, B2, C8, C9), the point (1, 2) can be taken as the top-left corner of the intercepted image, and the point (8, 9) can be taken as the bottom-right corner of the intercepted image. Since the intercepted image is a rectangle, the first to-be-matched block can be determined as a rectangle with a length of 7 cm, a height of 7 cm, a top-left corner coordinate of (1, 2), and a bottom-right corner coordinate of (8, 9).

[0128] In step 1032, the first block is matched with the corresponding first to-be-matched block to obtain a first matched block in the first to-be-matched block that matches the first block and a first matching degree between the first block and the first matched block.

[0129] In some embodiments, the matching of the first block with the corresponding first to-be-matched block in step 1032 to obtain the first matched block in the first to-be-matched block that matches the first block and the first matching degree between the first block and the first matched block can be implemented by the following technical solution: taking the size of the first block as the size of a search box, and performing traversal processing on the first to-be-matched block by using the search box to obtain at least one traversal block; when the number of the traversal blocks is one, the traversal block is taken as the first matched block that matches the first block, and the first matched block is compared with the first block to obtain the first matching degree between the first matched block and the first block; when the number of the traversal blocks is more than one, each traversal block is compared with the first block to obtain the first matching degree between the first block and each traversal block, and the traversal block with the highest first matching degree is taken as the first matched block that matches the first block.

[0130] As an example, before the matching processing, the block that matches the first block needs to be found from the first to-be-matched block. Since the size of the first block needs to be the same as the block to be matched during the matching processing, the size of the first block can be taken as the size of a search box, and the first to-be-matched block is traversed to obtain at least one traversal block.

[0131] As an example, since the size of the first block can be the same as the size of the first to-be-matched block, there is only one traversal block in the first to-be-matched block, which can be taken as the first matched block that matches the first block.

[0132] As an example, when the number of the traversal blocks is more than one, each traversal block can be compared with the first block to obtain the first matching degree between the first block and each traversal block, and the traversal block with the highest first matching degree is taken as the first matched block that matches the first block.

[0133] As an example, if there are traversal block A, traversal block B and traversal block C, it is determined that the first matching degree between traversal block A and the first block is 0.8, the first matching degree between traversal block B and the first block is 0.7, and the first matching degree between traversal block C and the first block is 0.9, and traversal block C is taken as the first matching block matched with the first block.

[0134] Through the above scheme, the block that is most matched with the first block can be found, and the accuracy of subsequent image matching is improved.

[0135] In some embodiments, the above comparison processing of each traversal block with the first block to obtain the first matching degree of each traversal block with the first block can be implemented through the following technical scheme: obtaining a first pixel at each pixel position in the first block; for each traversal block, the following processing is performed: obtaining a second pixel at each pixel position in the traversal block; for each pixel position, determining a pixel distance between the first pixel at the pixel position and the second pixel at the pixel position; performing fusion processing on the pixel distances of the plurality of pixel positions to obtain an overall pixel distance of the traversal block and the first block; and obtaining a first matching degree negatively correlated with the overall pixel distance.

[0136] As an example, first, the pixel value of each pixel position in the first block is determined, then the pixel value of each pixel position of the traversal block that needs to be compared is determined, and then the first matching degree between the first block and the traversal block is determined by comparing the pixel value of each pixel position.

[0137] As an example, the first pixel of each pixel position of the first block is:

[0138] [5, 5, 5

[0139] 5, 6, 7

[0140] 8, 9, 10]

[0141] The second pixel of each pixel position of the traversal block is:

[0142] [5, 5, 5

[0143] 6, 7, 8

[0144] 9, 7, 6]

[0145] Therefore, the pixel distance between the first pixel and the second pixel of each pixel position is:

[0146] [0, 0, 0

[0147] 1, 1, 1

[0148] 1, 2, 4]

[0149] The pixel distance of each pixel position can be fused, and the fusion can be summing the pixel distances, and the summing result is the overall pixel distance. In the above example, the overall pixel distance is 0+0+0+1+1+1+1+2+4, which is 10.

[0150] As an example, since the larger the overall pixel distance is, the greater the difference between the first block and the traversal block is, the overall pixel distance can be mathematically processed to obtain the first matching degree negatively correlated with the overall pixel distance.

[0151] As an example, the reciprocal of the overall pixel distance can be taken as the first matching degree, and the opposite number of the overall pixel distance can also be taken as the first matching degree. The mathematical processing of the overall pixel distance can be selected according to actual conditions, which is not limited here.

[0152] In step 104, the first difference degree between each first block and the corresponding first matching block is determined.

[0153] In some embodiments, the determination of the first difference degree between each first block and the corresponding first matching block in step 104 can be realized by the following technical scheme: for each first block, the following processing is performed: determining the first histogram of the first block and the second histogram of the first matching block corresponding to the first block; determining the correlation degree between the first histogram and the second histogram; obtaining the first difference degree negatively correlated with the correlation degree.

[0154] As an example, the pixel value of each pixel included in the first block can be counted, and a first histogram with the pixel value as the abscissa and the number of pixels corresponding to the pixel value as the ordinate can be drawn. Similarly, the pixel value of each pixel included in the first matching block can be counted, and a second histogram with the pixel value as the abscissa and the number of pixels corresponding to the pixel value as the ordinate can be drawn.

[0155] As an example, after the first histogram and the second histogram are determined, the correlation degree between the first histogram and the second histogram is determined, and then the first difference degree between the first block and the first matching block is determined.

[0156] As an example, the determination of the first histogram and the second histogram can adopt the correlation coefficient method, specifically, the correlation coefficient can measure the strength of linear correlation between two variables. If the correlation coefficient is close to 1, it means that the histograms are very similar; if it is close to -1, it means that the histograms are completely opposite; if it is close to 0, it means that there is no linear correlation. Various methods can be used to calculate the correlation coefficient, such as Pearson correlation coefficient, Spearman correlation coefficient, etc.

[0157] As an example, the way of determining the first histogram and the second histogram can also adopt mean square error, which can measure the average distance between the two histograms. The closer to zero, the more similar the histograms are.

[0158] As an example, since the higher the correlation, the more similar the first histogram and the second histogram are characterized, and the smaller the difference is, therefore the first difference degree and the similarity are negatively correlated. Therefore, after obtaining the similarity, the similarity can be mathematically processed to obtain the first difference degree.

[0159] As an example, the way of mathematically processing the similarity can be to take the reciprocal of the similarity as the first difference degree, or to take the opposite number of the similarity as the first difference degree. The specific way of mathematically processing the similarity can be selected according to actual conditions, which is not limited here.

[0160] Through the above scheme, by increasing the first difference degree in the process of determining the first block in the first image that exists with the content of the second image, the accuracy of the first block in the first image that exists with the content of the second image and has a difference is improved.

[0161] In step 105, based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, the first block in the first image that exists with the content of the second image and has a difference is determined.

[0162] In some embodiments, the determination of the first block in the first image that exists with the content of the second image and has a difference based on the first matching degree and the first difference degree between each first block and the corresponding first matching block in step 105 can be realized by the following technical scheme: for each first block, the following processing is performed: obtaining a difference score corresponding to the first matching degree positively correlated with the first block and the first difference degree negatively correlated with the first block; when the difference score is less than a difference score threshold, the first block is determined as the first block in the first image that exists with the content of the second image and has a difference.

[0163] As an example, after obtaining the first matching degree and the first difference degree, the first matching degree and the first difference degree can be mathematically processed to obtain a difference score positively correlated with the first matching degree and negatively correlated with the first difference degree.

[0164] As an example, the difference score can be determined by referring to the following formula (1):

[0165] C=A-B (1)

[0166] In formula (1), C is the difference score, A is the first matching degree, and B is the first difference degree. According to formula (1), the difference score can be obtained by subtracting the first difference degree from the first matching degree.

[0167] As an example, when the difference score is less than the difference score threshold, the difference of the first block is too large, and the first block can be determined as the first block in the first image that has the difference with the content of the second image.

[0168] As an example, if the difference score of the first block A is 0.6, and the difference score threshold is 0.7, the first block A can be determined as the first block in the first image that has the difference with the content of the second image.

[0169] As an example, referring to Figure 6 , Figure 6 is a schematic diagram of the first block in the first image that has the difference provided by the embodiment of the present application.

[0170] In Figure 6 , it can be known that “current selection: 1” is the first block that has the difference.

[0171] In some embodiments, after step 105 is performed, the technical solutions shown in steps 16 to 109 shown in Figure 3D may also be performed.

[0172] In step 106, the second image is subjected to target recognition-based block segmentation processing to obtain at least one second block of the second image.

[0173] As an example, by determining whether the first block in the first image has the difference with the second image, it can be determined that all the blocks in the first image can find the matching block in the second image. In order to ensure that all the blocks in the second image can find the matching block in the first image, the second image can be subjected to target recognition-based block segmentation processing.

[0174] As an example, the manner of subjecting the second image to target recognition-based block segmentation processing can refer to the process of subjecting the first image to target recognition-based block segmentation processing in step 102 described above, and will not be described in detail here.

[0175] In step 107, for each second block, the second block and the first image are subjected to matching processing to obtain a second matching block in the first image that matches the second block and a second matching degree between the second block and the second matching block.

[0176] As an example, the manner of subjecting the second block and the first image to matching processing to obtain the second matching block in the first image that matches the second block and the second matching degree between the second block and the second matching block can refer to the manner of subjecting the first block and the second image to matching processing to obtain the first matching block in the second image that matches the first block and the first matching degree between the first block and the first matching block shown in step 103 described above, and will not be described in detail here.

[0177] In some embodiments, the matching the second blocks with the first image in step 107 to obtain the second matching blocks in the first image matched with the second blocks and the second matching degrees between the second blocks and the second matching blocks can be implemented by the following technical solution: when at least one of the following conditions is met, the second blocks are matched with the first image to obtain the second matching blocks in the first image matched with the second blocks and the second matching degrees between the second blocks and the second matching blocks: the number of the first blocks is not greater than the number of the second blocks; the number of the first blocks having differences with the second image content in the first image is less than the first number threshold.

[0178] As an example, when the number of the first blocks is not greater than the number of the second blocks, it is represented that there may be blocks in the second blocks that are not matched with the first blocks, and when the number of the first blocks having differences with the second image content in the first image is less than the first number threshold, it is represented that there may be fewer blocks having differences in the first blocks, and the second blocks need to be matched.

[0179] In step 108, the second difference degrees between each second block and the corresponding second matching block are determined.

[0180] As an example, the manner of determining the second difference degrees between the second blocks and the corresponding second matching blocks can refer to the manner of determining the first difference degrees between the first matching blocks and the corresponding first matching blocks shown in step 104, which will not be described in detail here.

[0181] In step 109, based on the second matching degrees and the second difference degrees between each second block and the corresponding second matching block, the second blocks having differences with the first image content in the second image are determined.

[0182] As an example, the manner of determining the second blocks having differences with the first image content in the second image based on the second matching degrees and the second difference degrees between each second block and the corresponding second matching block can refer to the manner of determining the first blocks having differences with the second image content in the first image based on the first matching degrees and the first difference degrees between each first block and the corresponding first matching block shown in step 105, which will not be described in detail here.

[0183] Through the above scheme, the difference degrees of all the first blocks included in the first image and all the second blocks included in the second image can be determined, which ensures that all the blocks included in the first image and the second image are matched, and improves the accuracy of determining the difference blocks.

[0184] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0185] The image processing method provided by the embodiments of the present application can be applied to the automatic testing scene of game software and the automatic testing scene of shopping software. The image processing method provided by the embodiments of the present application is described taking the automatic testing scene of game software as an example.

[0186] Referring to Figure 7 , Figure 7 The flowchart in the actual application scene provided by the embodiments of the present application is shown.

[0187] Firstly, the width of the source image (first image) and the target image (second image) can be compared. When the widths of the two are inconsistent, the minimum value of the width of the source image and the width of the target image is taken as the standard width. The picture width of the image different from the standard width is scaled to the standard width, and the width-height ratio in the scaling process is kept unchanged.

[0188] Then, the height difference (size parameter difference) between the source image and the target image is obtained by comparing the height of the source image and the height of the target image, which is marked as gap. When the gap value is less than 100, the gap value is set to 100.

[0189] Then, a pre-trained image segmentation model is used to perform image segmentation on the source image and the target image respectively, and two groups of segmentation block data sourceBlock (first block) and targetBlock (second block) are obtained.

[0190] In the embodiments of the present application, the image segmentation model used for segmenting the image is not specifically limited and can be selected according to actual conditions. The specific image segmentation method can be image segmentation using YOLOv8, TensorFlow, etc.

[0191] The YOLOv8 framework is a deep learning framework for object detection, which uses convolutional neural networks (CNN) for object detection and can process video streams and detect moving objects in real time. The YOLOv8 framework provides flexible tools and modules, making it easy for researchers and developers to build and optimize their own object detection models. It has high performance, real-time performance and flexibility, and is suitable for various application scenarios such as autonomous driving and sports analysis.

[0192] The TensorFlow framework is an open-source framework for machine learning and deep learning. Through TensorFlow, developers can build, train and deploy various machine learning models. The core of TensorFlow is an abstract representation called "computation graph". The "computation graph" is composed of nodes and edges. Nodes represent mathematical operations, and edges represent the flow of data in the computation graph. This graph structure allows developers to express computation tasks in a high-level manner and can be easily extended to large-scale problems.

[0193] Traverse each first block of sourceBlock and the positioning coordinates (startXl, startYl, endXl, endYl) of the first block, wherein startXl is the horizontal coordinate of the top-left corner vertex of the first block, startYl is the vertical coordinate of the top-left corner vertex of the first block, endXl is the horizontal coordinate of the bottom-right corner vertex of the first block, and endYl is the vertical coordinate of the bottom-right corner vertex of the first block, and template match each block.

[0194] The process of template matching is described in detail below.

[0195] First, set the block to be matched in template matching as the corresponding block in the target image, and the coordinates of the block to be matched as startXl-gap, startYl-gap, endXl+gap, and endYl+gap, wherein startXl-gap is the horizontal coordinate of the top-left corner vertex of the block to be matched, startYl-gap is the vertical coordinate of the top-left corner vertex of the block to be matched, endXl+gap is the horizontal coordinate of the bottom-right corner of the block to be matched, and endYl+gap is the vertical coordinate of the bottom-right corner of the block to be matched (the coordinate system in which the source image and the target image are located is a coordinate system in which the positive direction of the X axis is to the right and the positive direction of the Y axis is downward). After template matching, the matched block and the first matching degree (matchDegree value) are obtained, and the coordinates of the matched block are (matchStartX, matchStartY, matchEndX, matchEndY), wherein matchStartX is the horizontal coordinate of the top-left corner vertex of the matched block, matchStartY is the vertical coordinate of the top-left corner vertex of the matched block, matchEndX is the horizontal coordinate of the bottom-right corner vertex of the matched block, and matchEndY is the vertical coordinate of the bottom-right corner vertex of the matched block.

[0196] Then, the matched block is cropped from the target image as the first matched block (matchImg), wherein the horizontal coordinate of the top-left corner vertex of the first matched block is matchStartX, the vertical coordinate of the top-left corner vertex of the first matched block is matchStartY, the horizontal coordinate of the bottom-right corner vertex of the first matched block is matchEndX, and the vertical coordinate of the bottom-right corner vertex of the first matched block is matchEndY. The first difference value (histDegree value) is obtained after histogram difference comparison between the first matched block and the cropped image of the first block (startXl, startYl, endXl, endYl) in the source image.

[0197] Traverse each first block in the first block, record the matchDegree value and the histDegree value corresponding to each first block, and obtain the traversal result (sourceData) of the first block.

[0198] After obtaining the sourceData in the first block set, in order to avoid the existence of the second block set which does not match the first block set, it is necessary to obtain the traversal result corresponding to the second block.

[0199] The process of obtaining the traversal result of the second block corresponding to the second block is described in detail below.

[0200] Traverse each second block of the targetBlock and the positioning coordinates (startX2, startY2, endX2, endY2) of the second block, wherein startX2 is the horizontal coordinate of the top left corner of the second block, startY2 is the vertical coordinate of the top left corner of the second block, endX2 is the horizontal coordinate of the bottom right corner of the second block, and endY2 is the vertical coordinate of the bottom right corner of the second block. Then, template matching can be performed on each second block.

[0201] First, set the block to be matched in template matching as the corresponding block in the source image, and the coordinates of the block to be matched as startX2-gap, startY2-gap, endX2+gap, and endY2+gap, wherein startX2-gap is the horizontal coordinate of the top left corner of the block to be matched, startY2-gap is the vertical coordinate of the top left corner of the block to be matched, endX2+gap is the horizontal coordinate of the bottom right corner of the block to be matched, and endY2+gap is the vertical coordinate of the bottom right corner of the block to be matched. After template matching, the matching block (matchStartX, matchStartY, matchEndX, matchEndY) and the second matching degree (matchDegree value) are obtained, wherein matchStartX is the horizontal coordinate of the top left corner of the matching block, matchStartY is the vertical coordinate of the top left corner, matchEndX is the horizontal coordinate of the bottom right corner of the matching block, and matchEndY is the vertical coordinate of the bottom right corner of the matching block.

[0202] Afterwards, the matching block is intercepted from the source image as a second matching block (matchImg), where the horizontal coordinate of the top-left corner of the second matching block is matchStartX, the vertical coordinate of the top-left corner of the second matching block is matchStartY, the horizontal coordinate of the bottom-right corner of the second matching block is matchEndX, and the vertical coordinate of the bottom-right corner of the second matching block is matchEndY. The second matching block is compared with the screenshot of the second block (startX2, startY2, endX2, endY2) in the source image in terms of histogram difference to obtain a second difference value (histDegre value).

[0203] Each second block in the second blocks is traversed, and the matchDegree value and the histDegree value corresponding to each second block are recorded to obtain a traversal result (targetData) of the second blocks.

[0204] After obtaining the traversal result of the first blocks and the traversal result of the second blocks, the sourceData and the targetData are traversed respectively. When the matchDegree-histDegree value of a block is less than 0.5 (which can be adjusted according to the scene in actual application), it is determined that the content of the block is different.

[0205] It can be understood that in the embodiments of the present application, user information and other related data are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0206] The following continues to describe an exemplary structure of the image processing apparatus 555 provided by the embodiments of the present application, which is implemented as a software module. In some embodiments, as shown in FIG. 5B, the software module stored in the image processing apparatus 555 of the memory 540 can include: Figure 2

[0207] A data acquisition module 5551 configured to acquire a first image and a second image.

[0208] A block segmentation module 5552 configured to perform target recognition-based block segmentation processing on the first image to obtain at least one first block of the first image.

[0209] An image matching module 5553 configured to, for each first block, perform matching processing on the first block and the second image to obtain a first matching block matched with the first block in the second image and a first matching degree between the first block and the first matching block.

[0210] ​The difference determination module 5554 is configured to determine a first difference between each of the first blocks and the corresponding first matching block.

[0211] The block determination module 5555 is configured to determine, based on the first matching degree and the first difference between each of the first blocks and the corresponding first matching block, a first block in the first image that has a difference from the content of the second image.

[0212] In some embodiments, the block segmentation module 5552 is further configured to perform target recognition processing on the first image to obtain a detection frame corresponding to each target recognition result; and perform segmentation processing on the first image based on at least one of the detection frames to obtain a first block corresponding to each of the detection frames.

[0213] In some embodiments, the image matching module 5553 is further configured to perform intercepting processing on the second image based on the first block to obtain a first to-be-matched block in the second image corresponding to each of the first blocks; and perform matching processing on the first block and the corresponding first to-be-matched block to obtain a first matching block in the first to-be-matched block that matches the first block and a first matching degree between the first block and the first matching block.

[0214] In some embodiments, the image matching module 5553 is further configured to determine a positioning coordinate of the first block in the first image; determine a query positioning coordinate based on the positioning coordinate of the first block in the first image; and perform intercepting processing on the second image based on the query positioning coordinate to obtain a first to-be-matched block in the second image corresponding to the first block.

[0215] In some embodiments, the image matching module 5553 is further configured to perform scaling processing on any of the first image and the second image to obtain a scaled first image or a scaled second image, wherein the scaled image has the same size as the image that has not been scaled; obtain a size parameter difference between the image before the scaling processing and the image after the scaling processing for the scaled image; and adjust the positioning coordinate of the first block in the first image based on the size parameter difference to obtain the query positioning coordinate.

[0216] In some embodiments, the image matching module 5553 is further configured to: take the size of the first block as a search box size, and traverse the first block using the search box to obtain at least one traversal block; when the number of the traversal blocks is one, take the traversal block as a first matching block matched with the first block, and compare the first matching block with the first block to obtain a first matching degree of the first matching block and the first block; when the number of the traversal blocks is more than one, compare each of the traversal blocks with the first block to obtain a first matching degree of each of the traversal blocks and the first block, and take the traversal block with the highest first matching degree as the first matching block matched with the first block.

[0217] In some embodiments, the image matching module 5553 is further configured to: obtain a first pixel at each pixel position in the first block; for each of the traversal blocks, perform the following processing: obtain a second pixel at each pixel position in the traversal block; for each of the pixel positions, determine a pixel distance between the first pixel at the pixel position and the second pixel at the pixel position; and fuse the pixel distances of the plurality of pixel positions to obtain an overall pixel distance of the traversal block and the first block; and obtain a first matching degree negatively related to the overall pixel distance.

[0218] In some embodiments, the difference degree determination module 5554 is further configured to, for each of the first blocks, perform the following processing: determine a first histogram of the first block and a second histogram of the first matching block corresponding to the first block; determine a correlation degree between the first histogram and the second histogram; and obtain a first difference degree negatively related to the correlation degree.

[0219] In some embodiments, the block determination module 5555 is further configured to, for each of the first blocks, perform the following processing: obtain a difference score positively related to the first matching degree corresponding to the first block and negatively related to the first difference degree corresponding to the first block; and when the difference score is less than a difference score threshold, determine the first block as a first block in the first image having a difference in content from the second image.

[0220] In some embodiments, the block determination module 5555 is further configured to perform target recognition based block segmentation on the second image to obtain at least one second block of the second image; for each second block, perform matching between the second block and the first image to obtain a second matching block in the first image that matches the second block and a second matching degree between the second block and the second matching block; determine a second difference degree between each second block and a corresponding second matching block; and determine a second block in the second image that has a difference in content from the first image based on the second matching degree and the second difference degree between each second block and a corresponding second matching block.

[0221] In some embodiments, the block determination module 5555 is further configured to perform matching between the second block and the first image to obtain a second matching block in the first image that matches the second block and a second matching degree between the second block and the second matching block when at least one of the following conditions is met: the number of first blocks is not greater than the number of second blocks; and the number of first blocks in the first image that have a difference in content from the second image is less than a first number threshold.

[0222] The embodiments of the present application provide a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions to cause the electronic device to perform the image processing method provided by the embodiments of the present application.

[0223] The embodiments of the present application provide a computer readable storage medium storing computer executable instructions, wherein the computer executable instructions or computer programs are stored, and when the computer executable instructions or computer programs are executed by a processor, the processor will execute the image processing method provided by the embodiments of the present application, for example, the image processing method shown in the above. Figure 3A

[0224] In some embodiments, the computer readable storage medium can be a RAM, a ROM, a flash memory, a magnetic surface memory, an optical disc, or a CD-ROM memory, etc. The computer readable storage medium can also be various devices including one or any combination of the above memories.

[0225] ​In some embodiments, the computer-executable instructions can take the form of programs, software, software modules, scripts, or code, written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and they can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0226] By way of example, a computer-executable instruction can, but need not, correspond to a file in a file system. A computer-executable instruction can be stored in one or more files, in a single file, or in multiple files, stored on one or more computer-readable media. A computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, or a computer-readable medium that can be accessed by a general purpose or special purpose computer system.

[0227] By way of example, a computer-executable instruction can be deployed to be executed on one electronic device or on multiple electronic devices that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0228] To sum up, the embodiments of the present application can achieve the following technical effects:

[0229] The first image and the second image are acquired, and the first image is subjected to block segmentation processing based on target recognition to obtain at least one first block of the first image. Since the block segmentation is based on target recognition, the useless area in the first image is removed at the same time the first block is obtained, thereby reducing the burden of subsequent matching processing and improving the efficiency of subsequent matching processing. Then, for each first block, the first block and the second image are subjected to matching processing to obtain a first matching block in the second image that matches the first block and a first matching degree between the first block and the first matching block. Through the matching processing, the first matching block corresponding to each first block can be obtained, thereby avoiding the case that the first block has no corresponding first matching block, and further improving the coverage rate of image detection. Then, the first difference degree between each first block and the corresponding first matching block can be determined, and based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, the first block in the first image that has a difference in content from the second image can be determined. The first difference degree can reflect the difference degree between the first block and the first matching block, and then the first block with a large difference degree is determined as the first block with a difference, thereby improving the recognition ability of the block with a difference and improving the accuracy of positioning the difference block.

[0230] The above merely provides an example of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement made within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized by, The method comprises: acquiring a first image and a second image; performing target recognition-based block segmentation processing on the first image to obtain at least one first block of the first image; for each first block, performing matching processing on the first block and the second image to obtain a first matching block in the second image that matches the first block and a first matching degree between the first block and the first matching block; determining a first difference degree between each first block and the corresponding first matching block; based on the first matching degree and the first difference degree between each first block and the corresponding first matching block, determining a first block in the first image that has a difference in content from the second image.

2. The method of claim 1, wherein, The target recognition-based block segmentation processing on the first image to obtain at least one first block of the first image comprises: performing target recognition processing on the first image to obtain a detection frame corresponding to each target recognition result; segmenting the first image based on at least one detection frame to obtain a first block corresponding to each detection frame.

3. The method of claim 1, wherein, The matching processing on the first block and the second image to obtain a first matching block in the second image that matches the first block and a first matching degree between the first block and the first matching block comprises: performing intercept processing on the second image based on the first block to obtain a first to-be-matched block in the second image corresponding to each first block; performing matching processing on the first block and the corresponding first to-be-matched block to obtain a first matching block in the first to-be-matched block that matches the first block and a first matching degree between the first block and the first matching block.

4. The method of claim 3, wherein, The intercept processing on the second image based on the first block to obtain a first to-be-matched block in the second image corresponding to each first block comprises: determining the positioning coordinates of the first block in the first image; based on the positioning coordinates of the first block in the first image, determining query positioning coordinates; based on the query positioning coordinates, performing intercept processing in the second image to obtain the first to-be-matched block in the second image corresponding to the first block.

5. The method of claim 4, wherein, The determination of the query positioning coordinates based on the positioning coordinates of the first block in the first image comprises: performing scaling processing on any of the first image and the second image to obtain a scaled first image or a scaled second image, wherein the scaled image has the same size as the unscaled image; for the scaled image, acquiring a size parameter difference between the image before the scaling processing and the image after the scaling processing; based on the size parameter difference, adjusting the positioning coordinates of the first block in the first image to obtain the query positioning coordinates.

6. The method of claim 3, wherein, The matching processing of the first block and the corresponding first to-be-matched block obtains a first matched block matched with the first block in the first to-be-matched block and a first matching degree between the first block and the first matched block, and includes the following steps. The size of the first block is taken as a search box size, and the search box is used to perform traversal processing in the first to-be-matched block to obtain at least one traversal block. When the number of the traversal blocks is one, the traversal block is taken as the first matched block matched with the first block, and the first matched block and the first block are compared to obtain the first matching degree between the first matched block and the first block. When the number of the traversal blocks is more than one, each of the traversal blocks is compared with the first block to obtain the first matching degree between each of the traversal blocks and the first block, and the traversal block with the highest first matching degree is taken as the first matched block matched with the first block.

7. The method of claim 6, wherein, The comparison processing of each of the traversal blocks with the first block to obtain the first matching degree between each of the traversal blocks and the first block includes the following steps. A first pixel of each pixel position in the first block is obtained. The following processing is performed for each of the traversal blocks. A second pixel of each of the pixel positions in the traversal block is obtained. For each of the pixel positions, a pixel distance between the first pixel of the pixel position and the second pixel of the pixel position is determined. Pixel distances of a plurality of the pixel positions are fused to obtain an overall pixel distance between the traversal block and the first block. A first matching degree negatively related to the overall pixel distance is obtained.

8. The method of claim 1, wherein, The determination of the first difference degree between each of the first blocks and the corresponding first matched block includes the following steps. The following processing is performed for each of the first blocks. A first histogram of the first block and a second histogram of the first matched block corresponding to the first block are determined. A correlation degree between the first histogram and the second histogram is determined. A first difference degree negatively related to the correlation degree is obtained.

9. The method of claim 1, wherein, The determination of the first block in the first image with a difference in content from the second image based on the first matching degree and the first difference degree between each of the first blocks and the corresponding first matched block includes the following steps. The following processing is performed for each of the first blocks. A difference score positively related to the first matching degree corresponding to the first block and negatively related to the first difference degree corresponding to the first block is obtained. When the difference score is less than a difference score threshold, the first block is determined as the first block in the first image with a difference in content from the second image.

10. The method of claim 1, wherein, The method further includes the following steps. The second image is subjected to target recognition-based block segmentation processing to obtain at least one second block of the second image. For each of the second blocks, matching processing of the second block and the first image is performed to obtain a second matched block in the first image matched with the second block and a second matching degree between the second block and the second matched block. determine a second difference degree between each of the second blocks and the corresponding second matched block; determine a second block in the second image that has a difference with the content of the first image based on the second matching degree and the second difference degree between each of the second blocks and the corresponding second matched block.

11. The method of claim 10, wherein, The matching processing of the second block and the first image to obtain the second matched block in the first image that matches the second block and the second matching degree between the second block and the second matched block includes: When at least one of the following conditions is met, the matching processing of the second block and the first image to obtain the second matched block in the first image that matches the second block and the second matching degree between the second block and the second matched block includes: The number of the first blocks is not greater than the number of the second blocks; The number of the first blocks in the first image that have a difference with the content of the second image is less than a first number threshold.

12. An image processing apparatus characterized by comprising: The apparatus includes: a data acquisition module configured to acquire a first image and a second image; a block segmentation module configured to perform target recognition-based block segmentation processing on the first image to obtain at least one first block of the first image; an image matching module configured to, for each of the first blocks, perform matching processing of the first block and the second image to obtain a first matched block in the second image that matches the first block and a first matching degree between the first block and the first matched block; a difference degree determination module configured to determine a first difference degree between each of the first blocks and the corresponding first matched block; a block determination module configured to determine a first block in the first image that has a difference with the content of the second image based on the first matching degree and the first difference degree between each of the first blocks and the corresponding first matched block.

13. An electronic device, comprising: The electronic device includes: a memory configured to store computer executable instructions; a processor configured to execute the computer executable instructions stored in the memory to implement the image processing method in any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions, the computer-executable instructions comprising: The computer executable instructions, when executed by the processor, implement the image processing method in any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the image processing method in any one of claims 1 to 11.

Images