Processing method and device and electronic equipment
The occluded foreground characters are processed through image processing technology, and a second image that does not affect the background is generated, which solves the occlusion problem in remote interaction and ensures clarity of information transmission and interactive experience.
Patent Information
- Application Number
- CN202411047865.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-19
AI Technical Summary
In remote interaction scenarios, the user's body or specific parts (such as hands) may inadvertently obscure the display content, causing the remote user to be unable to easily adjust the perspective angle to obtain obstructed information, affecting the information transmission efficiency and interactive experience.
Through image processing technology, based on the human segmentation algorithm in the field of deep learning, we identify and process the obscured foreground characters and generate a second image that does not affect the background part to ensure that remote users can clearly see the writing board and other content while retaining moderate Interactive information.
It effectively solves the problem of writing board occlusion, ensures smooth interaction between speakers and listeners, ensures clarity and accuracy of information transmission, and improves the communication efficiency and user experience of remote meetings.
Smart Images

Figure CN118982671A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a processing method, device and electronic equipment. Background Art
[0002] With the development of electronic technology, modern communication means have greatly promoted the ability of remote interaction between people. In the process of remote interaction using electronic devices, a common problem is that the user's body or specific parts (such as hands) may inadvertently block the display content behind it, such as the screen, electronic whiteboard, etc. This occlusion phenomenon poses a significant obstacle to remote users of remote interaction, because they cannot easily adjust their viewing angles to avoid occlusion like on-site participants, making it difficult to obtain the obscured information. This not only affects the efficiency of information transmission, but may also reduce the overall experience and effect of remote interaction. Summary of the invention
[0003] This application provides the following technical solutions:
[0004] On the one hand, the present application provides a processing method, comprising:
[0005] obtaining a first image;
[0006] Obtaining first interaction information;
[0007] Processing the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image;
[0008] The second image having the second interaction information is outputted.
[0009] In a possible implementation, processing the first image based on the first interaction information to obtain a second image having second interaction information includes one of the following:
[0010] generating second interaction information based on the first interaction information;
[0011] applying the second interaction information to the first image to generate a second image having the second interaction information;
[0012] or, adjusting the display attribute of the first interaction information to obtain adjusted first interaction information, and using the adjusted first interaction information as second interaction information to generate a second image having the second interaction information;
[0013] or, determining a current position of at least a portion of the first interaction information in the first image;
[0014] moving at least a portion of the first interaction information from the current location to a new location to obtain second interaction information;
[0015] The second interaction information is applied to the first image to generate a second image having the second interaction information.
[0016] In a possible implementation, obtaining first interaction information includes:
[0017] Determine a mask image corresponding to a target object based on the first image and an image frame before the first image; the values of pixels in the mask image belonging to the target object area are 0, and the values of pixels in the mask image belonging to the non-target object area are 255;
[0018] Based on the mask image, the first image is processed to obtain a first foreground image.
[0019] In a possible implementation, processing the first image based on the first interaction information to obtain a second image having second interaction information includes:
[0020] Based on the first image and the mask image, a background portion of the target image is updated to obtain an updated target image; the background portion comes from a background portion in at least one frame of image before the first image; the background portion in the at least one frame of image is different from a foreground portion in the at least one frame of image, and the foreground portion is an area corresponding to the target object;
[0021] Based on the mask image, processing the updated target image to obtain a second background image and a second foreground image;
[0022] Based on a first weighting factor corresponding to the first foreground image and a second weighting factor corresponding to the second foreground image, weighting the first foreground image and the second foreground image to obtain a foreground image to be used, wherein the first weighting factor is less than the second weighting factor;
[0023] A second image is generated based on the foreground image to be used and the second background image.
[0024] In a possible implementation, updating a background portion of a target image based on the first image and the mask image to obtain an updated target image includes:
[0025] Based on the mask image, the target image is processed to obtain a third background image and a third foreground image;
[0026] Based on the mask image, processing the first image to obtain a fourth background image and a fourth foreground image;
[0027] Based on a third weight factor corresponding to the third background image and a fourth weight factor corresponding to the fourth background image, weighting the third background image and the fourth background image to obtain a background image to be used, wherein the third weight factor is less than the fourth weight factor;
[0028] An updated target image is generated based on the background image to be used and the third foreground image.
[0029] In a possible implementation, processing the target image based on the mask image to obtain a third background image and a third foreground image includes:
[0030] Taking the mask image as a reference, modifying the pixel value of the area in the target image corresponding to the target object in the mask image to 0, so as to obtain a third background image;
[0031] Inverting the mask image to obtain a new mask image;
[0032] Taking the new mask image as a reference, the pixel values of the area in the target image corresponding to the non-target object in the new mask image are modified to 0, so as to obtain a third foreground image.
[0033] In a possible implementation, determining a mask image corresponding to the target object based on the first image and an image frame before the first image includes:
[0034] Converting the first image and an image frame before the first image into a first grayscale image and a second grayscale image;
[0035] Determine a difference between the first grayscale image and the second grayscale image to obtain a difference frame image;
[0036] The pixels whose pixel value difference in the difference frame image is less than a preset threshold are set to 255, and the pixels whose pixel value difference in the difference frame image is not less than the preset threshold are set to 0, so as to distinguish the background from the foreground;
[0037] Inputting the first image into a deep learning model to obtain an initial mask image corresponding to the target object determined by the deep learning model;
[0038] Taking the initial mask image as a reference, the region of the target object in the difference frame image is corrected to obtain a mask image corresponding to the target object.
[0039] In a possible implementation, obtaining a first image includes:
[0040] An image captured by an image acquisition device is obtained, and the image is used as a first image.
[0041] In another aspect of the present application, a processing device is provided, comprising:
[0042] A first acquisition module, used for acquiring a first image;
[0043] A second obtaining module, used to obtain first interaction information;
[0044] a third obtaining module, configured to process the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image;
[0045] An output module is used to output the second image with the second interaction information.
[0046] In a third aspect of the present application, an electronic device is provided, including:
[0047] A memory for storing at least one set of instructions;
[0048] The processor is used to call and execute the instruction set in the memory, and perform the following steps by executing the instruction set:
[0049] obtaining a first image;
[0050] Obtaining first interaction information;
[0051] Processing the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image;
[0052] The second image having the second interaction information is outputted. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0054] Figure 1 It is a flow chart of a processing method provided in Example 1 of the present application;
[0055] Figure 2It is a schematic diagram of an implementation scenario of a processing method provided by this application;
[0056] Figure 3 It is a flow chart of a processing method provided in Example 3 of the present application;
[0057] Figure 4 It is a flow chart of a processing method provided in Example 4 of the present application;
[0058] Figure 5 It is a schematic diagram of another implementation scenario of a processing method provided by the present application;
[0059] Figure 6 This is a schematic diagram of another implementation scenario of a processing method provided by the present application;
[0060] Figure 7 This is a schematic diagram of another implementation scenario of a processing method provided by the present application;
[0061] Figure 8 It is a structural schematic diagram of a processing device provided in this application. DETAILED DESCRIPTION
[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0063] In current conference scenarios, physical writing boards (such as physical whiteboards or physical blackboards) are still widely used as a traditional collaboration tool. However, with the rise of remote meetings, writing boards have exposed significant limitations in their application. Especially when the speaker is writing or explaining, his or her body or hands will inevitably block the content on the writing board. In remote meetings, it is difficult for remote listeners to obtain complete and clear information on the writing board because they cannot adjust the viewing angle or perspective.
[0064] An intuitive solution is to try to erase the person in the foreground through image processing technology to directly expose the obscured content of the writing board. Combined with the human segmentation algorithm in the field of deep learning, this function can be achieved in theory. However, this solution faces many challenges in practical applications. First, erasing the foreground person makes it difficult for the speaker to interact with the audience, because the audience cannot determine the specific content of their attention based on the speaker's gestures or instructions. For example, when the speaker points his finger at an important piece of information on the whiteboard, due to the absence of the foreground person, the audience may not be able to accurately understand the speaker's intention. Secondly, non-verbal communication methods such as gestures will also fail due to image processing, further reducing the communication efficiency of remote meetings.
[0065] Therefore, the present application provides a processing method that can effectively solve the problem of writing board obstruction and ensure smooth interaction between the speaker and the audience.
[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0067] Reference Figure 1 , is a flow chart of a processing method provided in Example 1 of the present application. The method can be applied to electronic devices. The present application does not limit the product type of the electronic device. For example, Figure 1 As shown, the method may include but is not limited to the following steps:
[0068] Step S101: Obtain a first image.
[0069] In this embodiment, the first image may include an image captured in a remote interaction scene. For example, in a remote conference, an image of a target object (e.g., a speaker) is captured, and the first image may include the target object and non-target objects (e.g., a writing board).
[0070] Obtaining a first image may include but is not limited to:
[0071] Step S1011, capturing an initial image in the remote interaction scene; the initial image includes a target object and non-target objects such as a writing board.
[0072] Step S1012: Calculate the homography transformation matrix according to the four corner points of the background plate in the initial image.
[0073] Step S1013: transform the initial image based on the homography transformation matrix to obtain a first image.
[0074] For example, Figure 2As shown in part (a), the initial image includes the target object, the writing board and other non-target objects. After the initial image is transformed based on the homography transformation matrix, the first image obtained is as shown in FIG. Figure 2 As shown in part (b), in the first image, the relative position between the target object and the background plate is reasonable.
[0075] Step S102: Obtain first interaction information.
[0076] In this embodiment, image content information in the first image can be acquired through image recognition technology, and first interaction information can be obtained based on the image content information, such as action information, gesture information, head posture information, etc. of the target object.
[0077] The first interaction information may affect the information presented by the first image. For example, the first interaction information may block part of the information presented by the first image.
[0078] Step S103: Process the first image based on the first interaction information to obtain a second image with second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image.
[0079] In a specific implementation of this embodiment, when the first image is processed based on the first interaction information, the background part in the first image may not be modified. The background part is different from the foreground part in the first image. The foreground part in the first image is the area corresponding to the target object in the first image.
[0080] During processing, the influence of the interactive information on the background portion in the first image may be reduced to ensure that the second interactive information does not excessively interfere with the background portion.
[0081] Step S104: output a second image with second interaction information.
[0082] In a remote interaction scenario, the second image with the second interaction information may be output to a remote electronic device connected to the electronic device.
[0083] Since the second interactive information in the second image has a weaker influence on the information presented in the second image, a user using a remote electronic device (ie, a remote user) can not only obtain relatively complete and clear background information from the second image, but also obtain appropriate interactive information.
[0084] In this embodiment, a first image is obtained, and first interaction information is obtained. The first interaction information may have an impact on the presentation information of the first image. To optimize the first image, the first image may be processed based on the first interaction information to obtain a second image with second interaction information. Compared with the first interaction information, the second interaction information has a weaker impact on the presentation information of the second image than the first interaction information. When the second image is output and presented to a remote user, it is ensured that the remote user can receive a second image that retains appropriate interaction information to support effective communication and reduces unnecessary interference. This process can not only ensure the clarity and accuracy of information transmission, but also maintain a smooth interaction experience between users participating in the interaction.
[0085] As another optional embodiment of the present application, a processing method is provided in Embodiment 2 of the present application. This embodiment is mainly an implementation method of step S103 in the processing method described in Embodiment 1 above. Step S103 may include but is not limited to:
[0086] Step S1031: Generate second interaction information based on the first interaction information.
[0087] In this embodiment, new interaction information may be generated based on the first interaction information. The new interaction information may be completely different from the first interaction information in form. The new interaction information is not merely obtained by adjusting the first interaction information, and the new interaction information may not retain any specific features of the first interaction information.
[0088] The new interaction information may be used as the second interaction information.
[0089] Step S1032: Apply the second interaction information to the first image to generate a second image with the second interaction information.
[0090] When the second interaction information is applied to the first image, the influence of the second interaction information on the image information in the first image that is influenced by the first interaction information should be reduced.
[0091] Step S103 may also include but is not limited to:
[0092] Step S1033: adjust the display attribute of the first interaction information to obtain adjusted first interaction information, and use the adjusted first interaction information as second interaction information to generate a second image having the second interaction information.
[0093] In this embodiment, the display attribute of the first interaction information is adjusted to obtain adjusted first interaction information, and the adjusted first interaction information has a weaker influence on the first image presentation information than the first interaction information.
[0094] The display attribute of the first interactive information may include but is not limited to: at least one of the transparency and display frequency of the first interactive information. For example, the transparency of the first interactive information may be increased to reduce the degree of occlusion of the first image presentation information by the first interactive information; the display frequency may be reduced, that is, the number of times the first interactive information appears in consecutive image frames may be reduced, so as to reduce the interference of the first interactive information on the user's line of sight and improve the user experience.
[0095] Step S103 may also include but is not limited to:
[0096] Step S1034: determine a current position of at least a portion of the first interaction information in the first image.
[0097] Step S1035: Move at least a portion of the first interaction information from the current position to a new position to obtain second interaction information.
[0098] At the new position, the visual interference or occlusion of the second interactive information on the information presented in the first image can be reduced.
[0099] In the process of adjusting the position of the first interaction information, the second interaction information may retain the same features as the first interaction information.
[0100] Step S1036: Apply the second interaction information to the first image to generate a second image having the second interaction information.
[0101] The second interactive information is applied to the first image to generate a second image having the second interactive information. The second image retains the main content and information of the first image and improves the overall readability and user experience of the second image by optimizing the position of the interactive information.
[0102] In the second image, the user can more clearly see the image content that may have been blocked or interfered with by the first interaction information, while still being able to notice the second interaction information, thereby ensuring an interactive experience.
[0103] As another optional embodiment of the present application, refer to Figure 3 , is a flow chart of a processing method provided in Example 3 of the present application, such as Figure 3 As shown, the method may include but is not limited to the following steps:
[0104] Step S201: Obtain a first image.
[0105] The detailed process of step S201 can refer to the relevant introduction of step S101 in embodiment 1, which will not be repeated here.
[0106] Step S202: Determine a mask image corresponding to the target object based on the first image and an image frame before the first image.
[0107] In this embodiment, a target detection algorithm may be used to identify a target object in the first image, and then the position and size of the target object in the first image may be determined to obtain a region of the target object.
[0108] In this embodiment, the difference between the first image and an image frame before the first image can be determined, and the difference is used as auxiliary information to adjust the area of the target object to eliminate errors caused by occlusion, deformation or false detection, thereby obtaining the adjusted area of the target object.
[0109] Based on the adjusted region of the target object, a mask image corresponding to the target object may be generated.
[0110] The first interaction information may come from the target object.
[0111] The size of the mask image can be the same as that of the first image, and each pixel in the mask image is assigned a specific value. In the mask image, the value of the pixel belonging to the target object area can be 0, and the value of the pixel belonging to the non-target object area can be 255.
[0112] Step S203: Process the first image based on the mask image to obtain a first foreground image.
[0113] The first foreground image may include a region of the target object in the first image.
[0114] Steps S202-S203 are an implementation of step S102 in Example 1.
[0115] Step S204: Generate a second image based on the mask image, the first foreground image and the first image.
[0116] The degree of influence of the region of the target object in the second image on the information presented in the second image is weaker than the degree of influence of the region of the target object in the first image on the information presented in the first image.
[0117] Step S204 is an implementation of step S103 in Example 1.
[0118] Step S205: output the second image.
[0119] Step S205 is an implementation of step S104 in Example 1.
[0120] In this embodiment, based on the first image and the previous frame of the first image, the mask image corresponding to the target object is determined, which can improve the accuracy and completeness of the target object area in the mask image and reduce false detection and missed detection. On this basis, since the accuracy of the mask image is guaranteed, the first foreground image obtained based on the mask image will also be able to accurately reflect the area of the target object. Based on the mask image, the first foreground image and the first image, the second image is determined. The process of generating the second image based on the mask image, the first foreground image and the first image may involve further processing of the first image. Through processing, it can be ensured that the area of the target object in the second image has a weaker influence on the information presented in the second image than the area of the target object in the first image has on the information presented in the first image, which can ensure the clarity and accuracy of information transmission and maintain a smooth interactive experience between users participating in the interaction.
[0121] As another optional embodiment of the present application, refer to Figure 4 , is a flow chart of a processing method provided in Example 4 of the present application. This example is mainly an implementation method of step S204 in Example 3. Figure 4 As shown, step S204 may include but is not limited to the following steps:
[0122] Step S2041: Based on the first image and the mask image, the background portion of the target image is updated to obtain an updated target image.
[0123] The background portion of the target image may come from a background portion in at least one frame of image before the first image.
[0124] The background portion of the target object can be updated through a continuous process, which is similar to step S2041 itself, but can be iterated according to new inputs (such as images and mask images of subsequent frames). This update can ensure the continuity and consistency of the background portion, especially when processing video sequences, it can reduce the background inconsistency problem caused by time changes.
[0125] The background portion in at least one frame of image is different from the foreground portion in at least one frame of image, and the foreground portion is an area corresponding to the target object.
[0126] When the background portion of the target object is updated, the foreground portion of the target image may not be updated.
[0127] Step S2042: Based on the mask image, the updated target image is processed to obtain a second background image and a second foreground image.
[0128] The second background image may include a region of the non-target object in the updated target image.
[0129] The second foreground image may include a region of the target object in the updated target image.
[0130] Step S2043: Based on a first weighting factor corresponding to the first foreground image and a second weighting factor corresponding to the second foreground image, weighted processing is performed on the first foreground image and the second foreground image to obtain a foreground image to be used, wherein the first weighting factor is smaller than the second weighting factor.
[0131] In this embodiment, each pixel value of the first foreground image can be multiplied by a first weight factor, and each pixel value of the second foreground image can be multiplied by a second weight factor. The corresponding pixels of the two weighted images are added to obtain a new image, that is, the foreground image to be used.
[0132] Since the second weight factor is greater than the first weight factor, the foreground image to be used can reflect more features of the second foreground image, but will also retain features of the first foreground image to a certain extent. For example, when the background portion of the target image is updated, if the updated target image no longer contains the target object, the second foreground image will not contain the target object. After weighted merging, since the second weight factor is larger, the transparency of the target object in the foreground image to be used will increase, and when the transparency of the target object increases, the background below the target object becomes visible.
[0133] The magnitudes of the first weight factor and the second weight factor can be set as required and are not limited in this application. However, it should be ensured that the first weight factor is smaller than the second weight factor.
[0134] Step S2044: Generate a second image based on the foreground image to be used and the second background image.
[0135] For example, in the second image, the area of the target object may become transparent or semi-transparent, and the information presented in the second image may be displayed through the transparent or semi-transparent area of the target object, thereby reducing interference to the user.
[0136] As another optional embodiment of the present application, a processing method is provided for embodiment 5 of the present application. This embodiment is mainly an implementation method of step S2041 in embodiment 4. Step S2041 may include but is not limited to the following steps:
[0137] Step S20411: Based on the mask image, the target image is processed to obtain a third background image and a third foreground image.
[0138] Corresponding to the mask image including the area of the target object and the area of the non-target object, after the target image is processed based on the mask image, the third background image may include image information of the area of the target image corresponding to the non-target object, and the third foreground image may include image information of the area of the target image corresponding to the target object.
[0139] The region corresponding to the target object in the target image may contain the target object or may not contain the target object.
[0140] Step S20412: Based on the mask image, process the first image to obtain a fourth background image and a fourth foreground image.
[0141] Corresponding to the mask image including the region of the target object and the region of the non-target object, after the first image is processed based on the mask image, the fourth background image may include image information of the region of the first image corresponding to the non-target object, and the fourth foreground image may include image information of the region of the first image corresponding to the target object. Corresponding to the implementation method in which the first image includes the target object, the region of the first image corresponding to the target object includes the target object.
[0142] Step S20413: Based on the third weight factor corresponding to the third background image and the fourth weight factor corresponding to the fourth background image, weighted processing is performed on the third background image and the fourth background image to obtain a background image to be used, and the third weight factor is smaller than the fourth weight factor.
[0143] In this embodiment, the pixel values of the third background image and the fourth background image may be weighted averaged according to their respective weight factors to obtain the background image to be used.
[0144] Since the fourth weight factor is larger, the fourth background image from the first image will contribute more to the final background, and the background image to be used will reflect more of the background features in the first image, so that it can contain the latest background information, while also retaining the background history of the target image to a certain extent. This processing method can make the background update reflect the latest visual content while maintaining a certain degree of continuity and stability.
[0145] The third weight factor and the fourth weight factor can be set as needed, and are not limited in this application. However, it should be ensured that the third weight factor is less than the fourth weight factor.
[0146] Step S20414: Generate an updated target image based on the background image to be used and the third foreground image.
[0147] In this embodiment, the updated target image combines the foreground portion of the target image, part of the background information of the target image, and part of the background information of the first image. That is, the updated target image can contain the latest background information, while also retaining the background history and foreground portion of the target image to a certain extent.
[0148] In this embodiment, as the target object moves, the mask image also changes accordingly. Accordingly, based on the mask image, the target image is processed to obtain a third background image and a third foreground image. The target object may no longer be included in the area corresponding to the target object in the third foreground image. For example, when the first frame of the video stream arrives, since there is no previous frame as a reference, the first frame can be regarded as the initial target image. When the second frame (which can be regarded as the first image) arrives, the mask image A corresponding to the target object can be determined based on the first image and the frame before the first image (i.e., the target image at this stage), as shown in FIG. Figure 5 As shown, the target image is processed based on the mask image A to obtain a third background image A11 and a third foreground image A22; the first image is processed based on the mask image A to obtain a fourth background image B11 and a fourth foreground image B22; based on a third weight factor (e.g., 0.2) corresponding to the third background image A11 and a fourth weight factor (e.g., 0.8) corresponding to the fourth background image B11, the third background image A11 and the fourth background image B11 are weighted to obtain a background image C1 to be used; based on the background image C1 to be used and the third foreground image A22, an updated target image D1 is generated.
[0149] As the target object moves, the mask image also changes accordingly. Accordingly, based on the mask image, the target image is processed to obtain a third background image and a third foreground image. The target object is no longer included in the area corresponding to the target object in the third foreground image. Accordingly, the updated target image no longer includes the target object. Figure 5 As shown, based on the third background image An1, the fourth background image Bn1 and the corresponding weight factors, the spare background image Cn to be used can be obtained, and based on the spare image cn to be used and the third foreground image An2, the updated target image Dn is generated, and the target image Dn no longer contains the target object.
[0150] It should be noted that in order to maintain the clarity and simplicity of the schematic diagram, Figure 5 The background image to be used is not shown.
[0151] If the updated target image no longer contains the target object, the updated target image is processed based on the mask image, and the obtained second foreground image also no longer contains the target object. The first foreground image and the second foreground image are weighted based on the first weight factor corresponding to the first foreground image and the second weight factor corresponding to the second foreground image to obtain the foreground image to be used. Since the first weight factor is less than the second weight factor, the target object in the foreground image to be used can become transparent and the transparency will increase. When the transparency of the target object increases, the second image is generated based on the foreground image to be used and the first background image, and the background under the target object in the second image becomes visible. For example, based on the first image and the previous frame of the first image, the mask image B corresponding to the target object is determined, and based on the mask image B, the first image is processed to obtain the first background image E11 and the first foreground image E22. Based on the first image and the mask image B, the background part of the target image is updated to obtain the updated target image, such as Figure 6 As shown, the updated target image no longer contains the target object. Based on the mask image B, the updated target image is processed to obtain the second background image F11 and the second foreground image F22. Based on the first weight factor 0.3 corresponding to the first foreground image E22 and the second weight factor 0.7 corresponding to the second foreground image F22, the first foreground image E22 and the second foreground image F22 are weighted to obtain the foreground image to be used. Based on the foreground image to be used and the second background image F11, the second image is generated. Figure 6 As shown, the background below the target object in the second image becomes visible.
[0152] As another optional embodiment of the present application, a processing method is provided for embodiment 6 of the present application. This embodiment is mainly an implementation method of step S20411 in embodiment 5. Step S20411 may include but is not limited to the following steps:
[0153] Step S11 , using the mask image as a reference, modifying the pixel value of the area in the target image corresponding to the target object in the mask image to 0, so as to obtain a third background image.
[0154] For example, Figure 5 As shown, the pixel value of the target object area in the mask image A is 0, and the pixel value of the remaining area is 255. Accordingly, the area corresponding to the target object in the mask image in the third background image A11 may appear black.
[0155] Step S12: invert the mask image to obtain a new mask image.
[0156] Inverting the mask image can be understood as exchanging the pixel values of the target object region and the pixel values of the non-target object region in the mask image.
[0157] The pixel values of the target object area in the new mask image are 255, and the pixel values of the non-target object area are 0.
[0158] Step S13: Using the new mask image as a reference, modify the pixel values of the area in the target image corresponding to the non-target object in the new mask image to 0, thereby obtaining a third foreground image.
[0159] In the third foreground image, the pixel values of the area corresponding to the target object in the new mask image are consistent with the pixel values of the corresponding area in the target image.
[0160] For example, Figure 5 As shown, the area in the third foreground image A22 corresponding to the non-target object in the new mask image may appear black.
[0161] The implementation methods of processing the target image based on the mask image to obtain the third background image and the third foreground image and processing the first image based on the mask image to obtain the fourth background image and the fourth foreground image are essentially similar. Therefore, the detailed process of step S20412 in embodiment 5 can refer to the relevant introduction of steps S11-S13, which will not be repeated here.
[0162] When elaborating on the image processing flow, the implementation method is essentially similar for the scenario of processing different target images (such as the second target image and the first image) based on a specific mask image to generate corresponding background and foreground image pairs. In view of this similarity, the detailed process of step S20412 described in Example 5 is highly consistent with steps S11 to S13. Therefore, in order to avoid repeated explanations and simplify the document, the specific implementation details of step S20412 can be directly referred to the detailed introduction of steps S11 to S13, and no additional explanation is given here.
[0163] As another optional embodiment of the present application, a processing method is provided for Embodiment 7 of the present application. This embodiment is mainly an implementation method of Step S202 in Embodiment 3. Step S202 may include but is not limited to the following steps:
[0164] Step S2021: converting the first image and an image frame before the first image into a first grayscale image and a second grayscale image.
[0165] Step S2022: determine the difference between the first grayscale image and the second grayscale image to obtain a difference frame image.
[0166] Step S2023, setting the pixels in the difference frame image whose pixel value difference is less than the preset threshold to 255, and setting the pixels in the difference frame image whose pixel value difference is not less than the preset threshold to 0, so as to distinguish the background from the foreground.
[0167] Step S2024: input the first image into the deep learning model to obtain an initial mask image corresponding to the target object determined by the deep learning model.
[0168] The value of the pixel in the target object area in the initial mask image may be 255, and the value of the pixel in the non-target object area may be 0.
[0169] Step S2025: Using the initial mask image as a reference, correct the region of the target object in the difference frame image to obtain a mask image corresponding to the target object.
[0170] In step S2025, each pixel originally marked as background (i.e., value 255) in the difference frame image is traversed to check whether these pixels are also located in the area marked as the target object in the initial mask image (i.e., pixels with a value of 255 in the initial mask image). If so, the values of these pixels in the difference frame image are forced to be 0, indicating that these locations should actually be part of the foreground, but were mistakenly classified as background in the preliminary background subtraction process.
[0171] For example, Figure 7 As shown, the pixels in the difference frame image whose pixel value difference is less than the preset threshold are set to 255, and the pixels in the difference frame image whose pixel value difference is not less than the preset threshold are set to 0, that is, the white part in the difference frame image is the background, and the black part is the foreground. The value of the pixels in the area of the target object in the initial mask image can be 255, and the value of the pixels in the area of the non-target object can be 0, that is, the white part in the initial mask image is the area of the target object, and the black part is the area of the non-target object. Traverse each pixel in the difference frame image that was originally marked as the background (that is, the value is 255), and check whether these pixels are also located in the area marked as the target object in the initial mask image (that is, the pixels with a value of 255 in the initial mask image). If so, force the values of these pixels in the difference frame image to be 0, and obtain the following Figure 7 The mask image is shown.
[0172] In this embodiment, the area of the target object in the difference frame image is corrected with reference to the initial mask image, which can ensure that the mask image corresponding to the target object finally obtained more accurately reflects the actual position and shape of the target object, thereby improving the accuracy and reliability of the mask image.
[0173] On the basis of improving the accuracy and reliability of the mask image, the foreground image and background image in the corresponding image can be accurately obtained, thereby effectively reducing the interference of the target object on the background in the second image. In addition, since the mask image can fully and clearly present the outline and details of the target object, the target object can be presented losslessly or with low loss in the second image.
[0174] As another optional embodiment of the present application, a processing method is provided for Embodiment 8 of the present application. This embodiment is mainly an implementation method of Step S101 in Embodiment 1. Step S101 may include but is not limited to the following steps:
[0175] Step S1014: obtaining an image captured by an image acquisition device, and using the image as a first image.
[0176] Through the image acquisition device, images can be acquired frame by frame, and each frame of the image is relatively independent. The image acquisition device can acquire images frame by frame, which is different from the real-time, continuous transmission of image streams in real-time communication scenarios (such as video calls). In real-time communication scenarios, images are transmitted as a continuous dynamic sequence. In step S1014, the focus is on the independent acquisition and processing of each frame of the image to ensure that each frame of the first image can achieve the expected effect.
[0177] Although the processing method described in Example 1 is mainly performed for frame-by-frame image acquisition, its flexibility and versatility also mean that it is also applicable to real-time communication scenarios. Through appropriate adjustment and optimization, the method can adapt to the processing requirements of real-time image streams, ensuring that high-quality image processing and effects can be maintained during continuous transmission.
[0178] Next, a processing device provided by the present application is introduced. The processing device introduced below and the processing method introduced above can refer to each other.
[0179] Reference Figure 8 The processing device includes: a first obtaining module 100, a second obtaining module 200, a third obtaining module 300 and an output module 400.
[0180] The first obtaining module 100 is used to obtain a first image.
[0181] The second obtaining module 200 is configured to obtain first interaction information.
[0182] The third obtaining module 300 is used to process the first image based on the first interaction information to obtain a second image with second interaction information, wherein the second interaction information has a weaker influence on the information presented by the second image than the first interaction information has on the information presented by the first image.
[0183] The output module 400 is configured to output the second image having the second interaction information.
[0184] The third obtaining module 300 may be specifically used for:
[0185] generating second interaction information based on the first interaction information;
[0186] applying the second interaction information to the first image to generate a second image having the second interaction information;
[0187] or, adjusting the display attribute of the first interaction information to obtain adjusted first interaction information, and using the adjusted first interaction information as second interaction information to generate a second image having the second interaction information;
[0188] or, determining a current position of at least a portion of the first interaction information in the first image;
[0189] Moving at least a portion of the first interaction information from a current location to a new location to obtain second interaction information;
[0190] The second interaction information is applied to the first image to generate a second image with the second interaction information.
[0191] The second obtaining module 200 may be specifically used for:
[0192] Based on the first image and an image frame before the first image, a mask image corresponding to the target object is determined; the values of pixels in the mask image belonging to the target object area are 0, and the values of pixels in the mask image belonging to the non-target object area are 255;
[0193] Based on the mask image, the first image is processed to obtain a first foreground image.
[0194] The third obtaining module 300 may be specifically used for:
[0195] Based on the first image and the mask image, a background portion of the target image is updated to obtain an updated target image; the background portion comes from a background portion in at least one frame of image before the first image; the background portion in at least one frame of image is different from a foreground portion in at least one frame of image, and the foreground portion is an area corresponding to the target object;
[0196] Based on the mask image, the updated target image is processed to obtain a second background image and a second foreground image;
[0197] Based on a first weighting factor corresponding to the first foreground image and a second weighting factor corresponding to the second foreground image, weighted processing is performed on the first foreground image and the second foreground image to obtain a foreground image to be used, wherein the first weighting factor is less than the second weighting factor;
[0198] A second image is generated based on the foreground image to be used and the second background image.
[0199] The third obtaining module 300 updates the background part of the target image based on the first image and the mask image to obtain the updated target image, which may specifically include:
[0200] Based on the mask image, the target image is processed to obtain a third background image and a third foreground image;
[0201] Based on the mask image, the first image is processed to obtain a fourth background image and a fourth foreground image;
[0202] Based on a third weight factor corresponding to the third background image and a fourth weight factor corresponding to the fourth background image, weighted processing is performed on the third background image and the fourth background image to obtain a background image to be used, wherein the third weight factor is less than the fourth weight factor;
[0203] An updated target image is generated based on the background image to be used and the third foreground image.
[0204] The third obtaining module 300 processes the target image based on the mask image to obtain the third background image and the third foreground image, which may specifically include:
[0205] Using the mask image as a reference, modify the pixel value of the area in the target image corresponding to the target object in the mask image to 0, so as to obtain a third background image;
[0206] Invert the mask image to obtain a new mask image;
[0207] With the new mask image as a reference, the pixel values of the area in the target image corresponding to the non-target object in the new mask image are modified to 0, thereby obtaining a third foreground image.
[0208] The process of the second obtaining module 200 determining the mask image corresponding to the target object based on the first image and an image frame before the first image may specifically include:
[0209] Converting the first image and an image frame before the first image into a first grayscale image and a second grayscale image;
[0210] Determine the difference between the first grayscale image and the second grayscale image to obtain a difference frame image;
[0211] The pixels whose pixel value difference in the difference frame image is less than the preset threshold are set to 255, and the pixels whose pixel value difference in the difference frame image is not less than the preset threshold are set to 0, so as to distinguish the background from the foreground;
[0212] Inputting the first image into the deep learning model to obtain an initial mask image corresponding to the target object determined by the deep learning model;
[0213] With the initial mask image as a reference, the region of the target object in the difference frame image is corrected to obtain the mask image corresponding to the target object.
[0214] The first obtaining module 100 may be specifically used for:
[0215] An image captured by an image acquisition device is obtained, and the image is used as a first image.
[0216] Corresponding to the above-mentioned processing method embodiment provided by the present application, the present application also provides an electronic device embodiment applying the processing method.
[0217] The electronic device may include the following structure:
[0218] Memory and processor.
[0219] A memory for storing at least one set of instructions;
[0220] The processor is used to call and execute the instruction set in the memory, and performs the following steps by executing the instruction set:
[0221] obtaining a first image;
[0222] Obtaining first interaction information;
[0223] Processing the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image;
[0224] A second image with second interaction information is output.
[0225] Corresponding to the above-mentioned processing method embodiment provided by the present application, the present application also provides an embodiment of a storage medium.
[0226] In this embodiment, the storage medium stores a computer program for implementing the processing method, and the computer program is executed by the processor to implement the following steps:
[0227] obtaining a first image;
[0228] Obtaining first interaction information;
[0229] Processing the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image;
[0230] A second image with second interaction information is output.
[0231] It should be noted that each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0232] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0233] For the convenience of description, the above device is described in terms of functions divided into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0234] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0235] The above is a detailed introduction to a processing method, device and electronic device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A processing method comprising: obtaining a first image; Obtaining first interaction information; Processing the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image; The second image having the second interaction information is outputted.
2. The processing method according to claim 1, wherein the first image is processed based on the first interaction information to obtain a second image having second interaction information, comprising: generating second interaction information based on the first interaction information; applying the second interaction information to the first image to generate a second image having the second interaction information; or, adjusting the display attribute of the first interaction information to obtain adjusted first interaction information, and using the adjusted first interaction information as second interaction information to generate a second image having the second interaction information; or, determining a current position of at least a portion of the first interaction information in the first image; moving at least a portion of the first interaction information from the current location to a new location to obtain second interaction information; The second interaction information is applied to the first image to generate a second image having the second interaction information.
3. The processing method according to claim 1, obtaining the first interaction information comprises: Determine a mask image corresponding to the target object based on the first image and an image frame before the first image; The value of the pixel in the mask image belonging to the target object area is 0, and the value of the pixel in the mask image belonging to the non-target object area is 255; Based on the mask image, the first image is processed to obtain a first foreground image.
4. The processing method according to claim 3, processing the first image based on the first interaction information to obtain a second image with second interaction information, comprising: Based on the first image and the mask image, updating the background portion of the target image to obtain an updated target image; The background portion comes from a background portion in at least one frame of image before the first image; The background portion in the at least one frame of image is different from the foreground portion in the at least one frame of image, and the foreground portion is the area corresponding to the target object; Based on the mask image, processing the updated target image to obtain a second background image and a second foreground image; Based on a first weighting factor corresponding to the first foreground image and a second weighting factor corresponding to the second foreground image, weighting the first foreground image and the second foreground image to obtain a foreground image to be used, wherein the first weighting factor is less than the second weighting factor; A second image is generated based on the foreground image to be used and the second background image.
5. The processing method according to claim 4, updating the background part of the target image based on the first image and the mask image to obtain an updated target image, comprising: Based on the mask image, the target image is processed to obtain a third background image and a third foreground image; Based on the mask image, processing the first image to obtain a fourth background image and a fourth foreground image; Based on a third weight factor corresponding to the third background image and a fourth weight factor corresponding to the fourth background image, weighting the third background image and the fourth background image to obtain a background image to be used, wherein the third weight factor is less than the fourth weight factor; An updated target image is generated based on the background image to be used and the third foreground image.
6. The processing method according to claim 5, processing the target image based on the mask image to obtain a third background image and a third foreground image, comprising: Taking the mask image as a reference, modifying the pixel value of the area in the target image corresponding to the target object in the mask image to 0, so as to obtain a third background image; Inverting the mask image to obtain a new mask image; Taking the new mask image as a reference, the pixel values of the area in the target image corresponding to the non-target object in the new mask image are modified to 0, so as to obtain a third foreground image.
7. The processing method according to claim 3, determining a mask image corresponding to the target object based on the first image and an image frame before the first image, comprising: Converting the first image and an image frame before the first image into a first grayscale image and a second grayscale image; Determine a difference between the first grayscale image and the second grayscale image to obtain a difference frame image; The pixels whose pixel value difference in the difference frame image is less than a preset threshold are set to 255, and the pixels whose pixel value difference in the difference frame image is not less than the preset threshold are set to 0, so as to distinguish the background from the foreground; Inputting the first image into a deep learning model to obtain an initial mask image corresponding to the target object determined by the deep learning model; Taking the initial mask image as a reference, the region of the target object in the difference frame image is corrected to obtain a mask image corresponding to the target object.
8. The processing method according to claim 1, obtaining the first image comprises: An image captured by an image acquisition device is obtained, and the image is used as a first image.
9. A processing device comprising: A first acquisition module, used for acquiring a first image; A second obtaining module, used to obtain first interaction information; a third obtaining module, configured to process the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image; An output module is used to output the second image with the second interaction information.
10. An electronic device comprising: A memory for storing at least one set of instructions; The processor is used to call and execute the instruction set in the memory, and perform the following steps by executing the instruction set: obtaining a first image; Obtaining first interaction information; Processing the first image based on the first interaction information to obtain a second image having second interaction information, wherein the second interaction information has a weaker influence on information presented by the second image than the first interaction information has on information presented by the first image; The second image having the second interaction information is outputted.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and storage medium
CN115497094A
Interaction method, electronic device, storage medium and program product
WO2023011296A1
