Image processing method and device, equipment, storage medium and program product

Abstract

The invention discloses an image processing method and device, equipment, a storage medium and a program product, relates to the technical field of communication, and can reduce visual interference and improve video display quality. The method comprises the following steps: performing visual complexity detection on a first video frame image to obtain first visual complexity information; determining a target area in the first video frame image according to the first visual complexity information; obtaining target visual display information of a target visual element in the target area; and based on the target visual display information, performing visual display weakening processing on a display area corresponding to the target visual element to obtain a second video frame image. The embodiment of the invention can reduce visual interference and improve the video display quality.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an image processing method, device, equipment, storage medium and program product. Background Art

[0002] For virtual scenes with multi-person interaction (e.g., in the field of game video processing), there are many methods for improving video quality and reducing visual noise. For example, some technologies use image processing algorithms to enhance or reduce noise in videos. In addition, there are also some technologies that focus on optimizing scene content, such as improving game rendering efficiency or optimizing game resource loading methods.

[0003] However, for some complex scenes, such as complex scenes where multiple visual elements (eg, virtual characters, skills, equipment, etc.) appear at the same time, these technologies often cannot effectively reduce visual interference. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, device, storage medium and program product, which can reduce visual interference and improve video display quality.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0006] Performing visual complexity detection on the first video frame image to obtain first visual complexity information;

[0007] Determining a target area in the first video frame image according to the first visual complexity information;

[0008] Acquire target visual display information of a target visual element in the target area;

[0009] Based on the target visual display information, a visual display weakening process is performed on a display area corresponding to the target visual element to obtain a second video frame image.

[0010] In a second aspect, an embodiment of the present application further provides an image processing device, including:

[0011] A first detection module, configured to perform visual complexity detection on a first video frame image to obtain first visual complexity information;

[0012] A first determining module, configured to determine a target area in the first video frame image according to the first visual complexity information;

[0013] A first acquisition module, used to acquire target visual display information of a target visual element in the target area;

[0014] The first processing module is used to perform visual display weakening processing on the display area corresponding to the target visual element based on the target visual display information to obtain a second video frame image.

[0015] In a third aspect, an embodiment of the present application further provides a communication device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the image processing method described above when executing the program.

[0016] In a fourth aspect, an embodiment of the present application further provides a readable storage medium, on which a program is stored, and when the program is executed by a processor, the steps in the image processing method as described above are implemented.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the image processing method as described above.

[0018] In the embodiment of the present application, by performing visual complexity detection on the first video frame image, first visual complexity information is obtained, and the target area can be determined in the first video frame image according to the first visual complexity information. Afterwards, based on the target visual display information of the target visual element in the target area, the display area corresponding to the target visual element can be subjected to visual display weakening processing to obtain a second video frame image, thereby reducing the visual interference of the target visual element and improving the video display quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the regions divided in the embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a coordinate system provided in an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0023] Figure 5 It is a structural diagram of the communication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the embodiments of the present application, the term "and / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0025] In the embodiments of the present application, the term "plurality" refers to two or more than two, and other quantifiers are similar.

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] See also Figure 1 , Figure 1 is a flow chart of the image processing method provided in the embodiment of the present application, such as Figure 1 As shown, the following steps are included:

[0028] Step 101: Perform visual complexity detection on a first video frame image to obtain first visual complexity information.

[0029] The first visual complexity information includes: the visual complexity of at least one visual element in the first video frame image.

[0030] The video frame images targeted by the embodiments of the present application include, but are not limited to, images of virtual scenes for multi-person interaction, such as video frame images of game videos, for example, video frame images of games with complex scenes in which multiple characters, multiple skills, and multiple equipment appear simultaneously.

[0031] Here, a machine learning model may be used to perform visual complexity detection on the first video frame image to obtain the first visual complexity information. Through the machine learning model, the position of the visual element in the image may also be determined at the same time. The visual complexity detection on the first video frame image includes color complexity, structural complexity and texture complexity of the visual elements of the first video frame image. Visual elements include but are not limited to virtual characters (such as game characters), skins, equipment, skills and maps.

[0032] The machine learning model is trained using at least one visual element of a training image and the visual complexity corresponding to at least one visual element. The machine learning model includes but is not limited to an artificial intelligence model, such as a convolutional neural network (CNN) or a transformer architecture, to process a large amount of game element training data.

[0033] The visual complexity of the visual elements of the training image is determined by the following information:

[0034] Color complexity, structural complexity, and texture complexity.

[0035] Among them, color complexity is used to analyze the color distribution and saturation of visual elements, structural complexity is used to evaluate the contour complexity and geometric shape of visual elements, and texture complexity is used to detect the texture and details of visual elements.

[0036] Taking a game video as an example, multiple video frame images of the game video are collected as training images. For the game video, all visual elements are collected, such as virtual characters, skins, equipment, skills, and maps. For each visual element, the visual complexity score (expressed as F) is performed through image processing algorithms and machine learning models.

[0037] The process of scoring visual complexity may include:

[0038] Calculation of color complexity Y: Assuming that a visual element has n colors, and the saturation of each color ranges from B1 to Bn (saturation is RGB saturation, ranging from 0% to 100%), then:

[0039]

[0040] Calculation of structural complexity J: This can be achieved by identifying and counting the number of objects that make up the visual elements, the number of contours of geometric shapes, the types of colors, etc.

[0041] J=X*n, where X represents the number of objects or the number of contours of geometric shapes or the types of colors that constitute the visual elements, and n can be set as needed, for example, a value of 10.

[0042] For example, for a certain virtual character, including a humanoid object and a weapon, then it is composed of 2 objects, and its corresponding structural complexity J=2*n (such as n=10). For a certain virtual character, it includes a humanoid object, a mount, and a weapon, then it is composed of 3 objects, and its corresponding structural complexity J=3*n. For a non-humanoid object, such as a skill, the number of continuous edge graphics it includes (that is, the number of geometric shape contours) can be identified. For example, the number of geometric shape contours is 3, and its corresponding structural complexity J=3*n. For skin, the types of colors it includes can be identified to obtain its corresponding structural complexity. For example, the true types of colors are 3, and its corresponding structural complexity J=3*n.

[0043] Calculation of texture complexity W: W = number of textures of the element * m (e.g. m = 10). The number of textures of the element can be obtained by identifying the image cutout technology.

[0044] Then, the visual complexity of each visual element is: F = Y*J*W.

[0045] After obtaining the visual complexity of each of the above-mentioned visual elements, training is performed using at least one visual element obtained and the visual complexity corresponding to at least one visual element to obtain a machine learning model to process a large amount of training data of game elements.

[0046] Step 102: Determine a target area in the first video frame image according to the first visual complexity information.

[0047] In this step, the first video frame image is divided into a plurality of regions, and second visual complexity information of each region is determined according to the first visual complexity information. Then, a target region is determined from the plurality of regions according to the second visual complexity information.

[0048] The number of regions into which the first video frame image is divided can be set as required. Figure 2 As shown, the first video frame image is divided into 20 regions. For each of the regions, the sum of the visual complexity of the visual elements in each of the regions is taken as the second visual complexity information (Fw) of each of the regions.

[0049] If the second visual complexity information of the first area is greater than the first threshold and all visual elements in the first area are located in the first area, the first area is taken as the target area; or if the second visual complexity information of the first area is greater than the first threshold and some or all visual elements in the first area are located in the first area and other areas at the same time, the first area and the other areas are taken as the target areas; wherein the first area is any area of ​​the multiple areas. The first threshold is set as required.

[0050] That is to say, if a certain visual element is only located in the first area, the first area is the target area; if a certain visual element is partially located in the first area and partially located in other areas, the first area and other areas are both the target areas. In this way, all areas related to the visual element can be processed, thereby improving the quality of the obtained image.

[0051] Step 103: Acquire target visual display information of the target visual element in the target area.

[0052] The target visual display information includes the target visual complexity and target influence weight corresponding to the target visual element, and the target visual element includes a virtual character. Since the virtual character is the information that the user cares about most, the target visual element including the virtual character is processed here. The target visual complexity corresponding to the target visual element is obtained by the above method, and the target influence weight is obtained by the following method:

[0053] The target influence weight of the target visual element is determined according to the virtual character weight of at least one virtual character in a first display area and the display ratio of the at least one virtual character in the first display area, wherein the first display area is used to display the target visual element.

[0054] Specifically, the display ratio of each virtual character in the first display area refers to the ratio of the portion of the virtual character displayed in the first display area to the overall size of the virtual character. If half of a virtual character is displayed in the first display area, then the display ratio is 50%. Different display ratios may correspond to different weight values, which may be set as needed. For example, if the display ratio is less than 50%, the corresponding weight is 0.2.

[0055] For example, in the display area of ​​visual element A1, if less than 50% of a virtual character is located in the display area, then the contribution weight of this virtual character to this visual element is Q*0.2; if more than 50% but less than 100% of a virtual character is located in the display area, then the contribution weight of this virtual character to this visual element is Q*0.5; if the entire virtual character is located in the display area, then the contribution weight of this virtual character to this visual element is Q.

[0056] If there are three virtual characters in the first display area, of which two virtual characters are completely located in the first display area and one virtual character has less than 50% of its part located in the first display area, then the target influence weight Qn=Q+Q+0.2Q=2.2Q.

[0057] Step 104: Based on the target visual display information, a visual display weakening process is performed on the display area corresponding to the target visual element to obtain a second video frame image.

[0058] For all target areas, the average value of the visual complexity of the visual elements in the target area is used as the third threshold Fp, and the average value of the game weights of the visual elements in the target area is used as the fourth threshold.

[0059] To this end, we can establish Figure 3 According to the third and fourth thresholds mentioned above, Figure 3 The coordinate system shown is divided into four components: 31, high game impact, low visual complexity; 32, high game impact, high visual complexity; 33, low game impact, low visual complexity; 34, low game impact, high visual complexity. For user experience, here, the elements with high game impact and high visual complexity are kept unchanged (that is, 33), and other visual primary colors are processed.

[0060] Specifically, if the target visual complexity of the target visual element is less than a third threshold, or the target visual complexity of the target visual element is greater than the third threshold and the target influence weight of the virtual character corresponding to the target visual element is less than a fourth threshold, a candidate image corresponding to the target visual element is obtained by cutout processing. The candidate image is then subjected to visual display weakening processing to obtain a target image, wherein the purpose of the visual display weakening processing is to reduce saturation and weaken display effects, such as black and white processing, stroke processing, etc. Afterwards, the target image is synthesized with images other than the candidate image in the first video frame image to obtain the second video frame image. During the synthesis, the target image is placed at the position of the candidate image in the first video frame image, thereby ensuring the effect of the second video frame image obtained.

[0061] That is, reference Figure 3 For the display area corresponding to a target visual element, after analysis, if the target visual complexity of the target visual element and the target influence weight of the virtual character corresponding to the target visual element are in the three intervals of 31, 33 and 34, then it is necessary to weaken the visual display thereof.

[0062] The adjusted and optimized game elements will be synthesized into new video frames, and the regenerated game video will have reduced visual complexity, thereby improving the audience's viewing experience.

[0063] In the embodiment of the present application, by performing visual complexity detection on the first video frame image, first visual complexity information is obtained, and the target area can be determined in the first video frame image according to the first visual complexity information. Afterwards, based on the target visual display information of the target visual element in the target area, the display area corresponding to the target visual element can be subjected to visual display weakening processing to obtain a second video frame image, thereby reducing the visual interference of the target visual element and improving the video display quality.

[0064] See also Figure 4 , Figure 4 is a structural diagram of an image processing device provided in an embodiment of the present application. Figure 4 As shown, the image processing device includes:

[0065] The first detection module 401 is used to perform visual complexity detection on the first video frame image to obtain first visual complexity information; the first determination module 402 is used to determine the target area in the first video frame image according to the first visual complexity information; the first acquisition module 403 is used to obtain target visual display information of the target visual element in the target area; the first processing module 404 is used to perform visual display weakening processing on the display area corresponding to the target visual element based on the target visual display information to obtain the second video frame image.

[0066] Optionally, the first detection module 401 is further used to perform visual complexity detection on the first video frame image using a machine learning model to obtain the first visual complexity information;

[0067] The machine learning model is obtained by training using at least one visual element of a training image and the visual complexity corresponding to the at least one visual element;

[0068] The visual complexity of the visual elements of the training image is determined by the following information:

[0069] Color complexity, structural complexity, and texture complexity.

[0070] Optionally, the first determining module 402 includes:

[0071] A division submodule, used for dividing the first video frame image into a plurality of regions;

[0072] a determination submodule, configured to determine second visual complexity information of each of the regions according to the first visual complexity information;

[0073] A selection submodule is used to determine a target area from the multiple areas according to the second visual complexity information.

[0074] Optionally, the determination submodule is used to, for each of the regions, take the sum of the visual complexities of the visual elements in each of the regions as the second visual complexity information of each of the regions.

[0075] Optionally, the selection submodule is used to use the first area as the target area if the second visual complexity information of the first area is greater than a first threshold and all visual elements in the first area are located in the first area; or

[0076] If the second visual complexity information of the first area is greater than the first threshold and part or all of the visual elements in the first area are located in the first area and other areas at the same time, the first area and the other areas are used as the target areas;

[0077] The first area is any area among the multiple areas.

[0078] Optionally, the target visual display information includes a target visual complexity and a target influence weight corresponding to the target visual element, and the target visual element includes a virtual character;

[0079] The target impact weight is obtained in the following way:

[0080] The target influence weight of the target visual element is determined according to the virtual character weight of at least one virtual character in a first display area and the display ratio of the at least one virtual character in the first display area, wherein the first display area is used to display the target visual element.

[0081] Optionally, the first processing module 404 includes:

[0082] A first processing submodule is configured to obtain a candidate image corresponding to the target visual element through a cutout process if the target visual complexity of the target visual element is less than a third threshold, or if the target visual complexity of the target visual element is greater than the third threshold and the target influence weight of the virtual character corresponding to the target visual element is less than a fourth threshold;

[0083] A second processing submodule is used to perform visual display weakening processing on the candidate image to obtain a target image;

[0084] A third processing submodule, configured to synthesize the target image with images other than the candidate image in the first video frame image to obtain the second video frame image;

[0085] The third threshold is the average value of the visual complexity of the visual elements in the target area, and the fourth threshold is the average value of the game weights of the visual elements in the target area.

[0086] The device provided in the embodiment of the present application can execute the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0087] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0089] See also Figure 5 An embodiment of the present application provides a communication device, comprising: a memory 501, a processor 502, and a program stored in the memory 501 and executable on the processor 502; the processor 502 is used to read the program in the memory 501 to implement the steps in the image processing method described above.

[0090] The embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by the processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it is not repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0091] The embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

[0092] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0093] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0094] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Performing visual complexity detection on the first video frame image to obtain first visual complexity information; Determining a target area in the first video frame image according to the first visual complexity information; Acquire target visual display information of a target visual element in the target area; Based on the target visual display information, a visual display weakening process is performed on a display area corresponding to the target visual element to obtain a second video frame image.

2. The method according to claim 1, characterized in that The performing visual complexity detection on the first video frame image to obtain first visual complexity information includes: Performing visual complexity detection on the first video frame image using a machine learning model to obtain the first visual complexity information; The machine learning model is obtained by training using at least one visual element of a training image and the visual complexity corresponding to the at least one visual element; The visual complexity of the visual elements of the training image is determined by the following information: Color complexity, structural complexity, and texture complexity.

3. The method according to claim 1 or 2, characterized in that: The determining a target area in the first video frame image according to the first visual complexity information includes: Dividing the first video frame image into a plurality of regions; Determining second visual complexity information of each of the regions respectively according to the first visual complexity information; determining a target area from the multiple areas according to the second visual complexity information; For each of the regions, the sum of the visual complexities of the visual elements in each of the regions is used as the second visual complexity information of each of the regions.

4. The method according to claim 3, characterized in that The determining a target area from the multiple areas according to the second visual complexity information comprises: If the second visual complexity information of the first region is greater than the first threshold and all visual elements in the first region are located in the first region, the first region is used as the target region; or If the second visual complexity information of the first area is greater than the first threshold and part or all of the visual elements in the first area are located in the first area and other areas at the same time, the first area and the other areas are used as the target areas; The first area is any area among the multiple areas.

5. The method according to claim 1, characterized in that The target visual display information includes the target visual complexity and the target influence weight corresponding to the target visual element, and the target visual element includes a virtual character; The target impact weight is obtained in the following way: The target influence weight of the target visual element is determined according to the virtual character weight of at least one virtual character in a first display area and the display ratio of the at least one virtual character in the first display area, wherein the first display area is used to display the target visual element.

6. The method according to claim 5, characterized in that The step of performing visual display weakening processing on a display area corresponding to the target visual element based on the target visual display information to obtain a second video frame image includes: If the target visual complexity of the target visual element is less than the third threshold, or the target visual complexity of the target visual element is greater than the third threshold and the target influence weight of the virtual character corresponding to the target visual element is less than the fourth threshold, a candidate image corresponding to the target visual element is obtained by cutout processing; Perform visual display weakening processing on the candidate image to obtain a target image; synthesizing the target image with images other than the candidate image in the first video frame image to obtain the second video frame image; The third threshold is the average value of the visual complexity of the visual elements in the target area, and the fourth threshold is the average value of the game weights of the visual elements in the target area.

7. An image processing device, characterized in that: include: A first detection module, configured to perform visual complexity detection on a first video frame image to obtain first visual complexity information; A first determining module, configured to determine a target area in the first video frame image according to the first visual complexity information; A first acquisition module, used to acquire target visual display information of a target visual element in the target area; The first processing module is used to perform visual display weakening processing on the display area corresponding to the target visual element based on the target visual display information to obtain a second video frame image.

8. A communication device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps of the image processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps in the image processing method according to any one of claims 1 to 6.

Images