Image desensitization method and related device
By identifying the outer contour curve of the content to be desensitized in the image and generating the same type but different styles of desensitized content, replacing the content to be desensitized, the problem of image quality desensitization in the prior art is solved, and the user experience is improved and image aesthetics.
Patent Information
- Application Number
- CN202211049570.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-08-30
AI Technical Summary
In the existing image desensitization technology, the mosaic smearing method is too rough, destroying image quality, reducing image aesthetics, and affecting user experience.
By determining the outer contour curve of the content to be desensitized, a desensitized content with a shape that conforms to the outer contour curve is generated, and a replacement of the content to be desensitized, ensuring that the desensitized content is the same as the original content type but the image style is different, and avoiding the desensitized content that is not desensitized Image quality.
Desensitization processing without affecting the image quality is achieved, increasing the aesthetics and fun of the image and improving the user experience.
Smart Images

Figure CN115408710B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to an image desensitization method and related devices. Background Art
[0002] Image desensitization is a technology that removes desensitized content such as faces and license plates from images. In the fields of intelligent transportation and security, video images that may contain content to be desensitized are collected through cameras, and image desensitization technology is needed to prevent the leakage of desensitized content and provide a safe use environment.
[0003] In the related technology, the content to be desensitized is taken as the detection target, and the area of the content to be desensitized is detected from the image to be desensitized through the target detection task, and then the area is mosaicked by mosaic or other means, so as to achieve the purpose of making the content to be desensitized invisible, thereby realizing image desensitization.
[0004] However, this mosaic smearing method is too rough and will destroy the image information of some areas where the non-desensitized content is located, reducing the image quality, thereby greatly reducing the aesthetics of the image and affecting the user experience. Summary of the invention
[0005] In order to solve the above technical problems, the present application provides an image desensitization method and related devices to obtain desensitized images that do not affect the image quality, and different image styles can bring interest to desensitization and improve the user experience.
[0006] The embodiments of the present application disclose the following technical solutions:
[0007] On the one hand, an embodiment of the present application provides an image desensitization method, the method comprising:
[0008] Acquire an image to be desensitized, wherein the image to be desensitized includes content to be desensitized;
[0009] Determine an outer contour curve of the outer contour of the to-be-desensitized content in the to-be-desensitized image;
[0010] According to the outer contour curve, generating desensitized content whose shape conforms to the outer contour curve, wherein the desensitized content and the content to be desensitized have the same content type and different image styles;
[0011] A desensitized image is obtained by replacing the content to be desensitized in the image to be desensitized with the desensitized content.
[0012] On the other hand, an embodiment of the present application provides an image desensitization device, the device comprising an acquisition unit, a determination unit, a generation unit and a replacement unit:
[0013] The acquisition unit is used to acquire the image to be desensitized, wherein the image to be desensitized includes the content to be desensitized;
[0014] The determining unit is used to determine an outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized;
[0015] The generating unit is configured to generate, according to the outer contour curve, desensitized content having a shape conforming to the outer contour curve, wherein the desensitized content and the content to be desensitized have the same content type and a different image style;
[0016] The replacement unit is used to obtain a desensitized image by replacing the to-be-desensitized content in the to-be-desensitized image with the desensitized content.
[0017] On the other hand, an embodiment of the present application provides an image desensitization method, the method comprising:
[0018] Acquiring a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle;
[0019] When it is determined that the vehicle environment image includes content to be desensitized, generating corresponding desensitized content according to the content to be desensitized, wherein the desensitized content has the same content type and a different image style as the content to be desensitized;
[0020] Obtaining a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content;
[0021] In response to an acquisition request sent by an external device for the vehicle environment image, the desensitized environment image is returned to the external device.
[0022] On the other hand, an embodiment of the present application provides an image desensitization device, the device comprising an acquisition unit, a generation unit, a replacement unit and a sending unit:
[0023] The acquisition unit is used to acquire a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle;
[0024] The generating unit is configured to generate corresponding desensitized content according to the content to be desensitized when it is determined that the vehicle environment image includes the content to be desensitized, wherein the desensitized content has the same content type and a different image style as the content to be desensitized;
[0025] The replacing unit is used to obtain a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content;
[0026] The sending unit is used to return the desensitized environment image to the external device in response to an acquisition request sent by the external device for the vehicle environment image.
[0027] On the other hand, an embodiment of the present application provides a computer device, the computer device comprising a processor and a memory:
[0028] The memory is used to store program code and transmit the program code to the processor;
[0029] The processor is configured to execute the method described above according to the instructions in the program code.
[0030] On the other hand, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method described in the above aspects.
[0031] On the other hand, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method described above.
[0032] It can be seen from the above technical solution that when obtaining an image to be desensitized including content to be desensitized, in order to protect the safety of the content to be desensitized, the image to be desensitized can be subjected to image desensitization. The outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized can be determined, and the desensitized content whose shape conforms to the outer contour curve is generated. Since the outer contour curve can accurately distinguish the boundary between the content to be desensitized and the non-content to be desensitized in the image to be desensitized, the image quality of the non-content to be desensitized in the image to be desensitized will not be destroyed by replacing the content to be desensitized in the image to be desensitized with the desensitized content. Moreover, the desensitized content and the content to be desensitized have the same content type and different image styles. The same content type makes the desensitized content in the image position of the original content to be desensitized, and the non-content to be desensitized in the image to be desensitized is not abrupt in the image as a whole, and the desensitized content can establish the original content association with the content to be desensitized with the non-content to be desensitized content, and the different image styles can reflect the difference in style between the desensitized content and the non-content to be desensitized content, effectively prompting the user that this is the desensitized content, rather than the original content in the image to be desensitized, to avoid misleading. Therefore, the desensitized content generated by the outer contour curve of the content to be desensitized can effectively realize image desensitization and obtain a desensitized image that does not affect the image quality. Moreover, different image styles can bring fun to desensitization and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0034] Figure 1 A schematic diagram of an image desensitization scenario provided in an embodiment of the present application;
[0035] Figure 2 A method flow chart of an image desensitization method provided in an embodiment of the present application;
[0036] Figure 3 A schematic diagram of a method for extracting a facial contour curve provided in an embodiment of the present application;
[0037] Figure 4 A schematic diagram of a profile-desensitized sample pair provided in an embodiment of the present application;
[0038] Figure 5 A schematic diagram of a network structure for training a content generation model provided in an embodiment of the present application;
[0039] Figure 6 A schematic diagram of a network structure for training a style transfer model provided in an embodiment of the present application;
[0040] Figure 7 A method for applying to an embodiment of the present application is provided Figure 5-Figure 6 Schematic diagram of the generator network structure in ;
[0041] Figure 8 A method for applying to an embodiment of the present application is provided Figure 5-Figure 6 Schematic diagram of the discriminator network structure in ;
[0042] Fig. 9 A schematic diagram of images involved in a training data set provided in an embodiment of the present application;
[0043] Fig.10 A flow chart of a face desensitization method for replacing a real face with an animated face provided in an embodiment of the present application;
[0044] Fig.11 A schematic diagram of a face desensitization scenario in which a real face is replaced by an animated face provided in an embodiment of the present application;
[0045] Fig.12 A schematic diagram of the network structure of a face detection model provided in an embodiment of the present application;
[0046] Fig.13 A method flow chart of another image desensitization method provided in an embodiment of the present application;
[0047] Fig.14 A device structure diagram of an image desensitization device provided in an embodiment of the present application;
[0048] Fig.15 A device structure diagram of another image desensitization device provided in an embodiment of the present application;
[0049] Fig.16 A structural diagram of a terminal device provided in an embodiment of the present application;
[0050] Fig.17 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings.
[0052] Since in many fields, video images that may include personal content to be desensitized are collected by camera devices, image desensitization technology is needed to avoid the leakage of desensitized content and provide users with a safe usage environment.
[0053] In the related art, the part of the image involving the content to be desensitized is mainly processed by mosaic smearing, which will destroy the image information of some non-privacy areas, seriously reduce the image quality, and reduce the aesthetics of the image, affecting the user's experience.
[0054] To this end, the embodiments of the present application provide an image desensitization method and related devices to obtain a desensitized image that does not affect the image quality, and different image styles can bring interest to the desensitization and improve the user experience.
[0055] That is to say, compared with the related art, the embodiment of the present application provides a friendly desensitization: while removing the privacy information or sensitive information in the image, it tries not to destroy other area information in the image that is not related to the privacy information, and tries to maintain the aesthetics of the image. This friendly desensitization can be applied to a large number of smart transportation and security scenarios, for example, it can be applied to at least the following scenarios:
[0056] Training data return:
[0057] Autonomous driving and advanced driver assistance require the use of image perception algorithms to detect various targets in driving scenes, such as pedestrians, vehicles, lane lines, traffic signs, traffic lights, drivable areas, etc. The development and iteration of these perception algorithms require a large amount of image data. If we only rely on dedicated image data collection vehicles to collect data, it is obviously not enough to meet the needs. At this time, images taken by mass-produced cars on the market can be transmitted back. Since there are many mass-produced cars and they are widely distributed, there is a strong guarantee for the quantity and diversity of image data.
[0058] The images sent back from the vehicle will contain private information such as faces, and need to be desensitized first. If the method of coding or smearing the rectangular frame in the related art is used, it is not conducive to the training of the perception model. However, the image desensitization method proposed in this application can achieve desensitization while greatly avoiding image damage, which is friendly to model training.
[0059] Parking Sentry Mode:
[0060] When the vehicle is parked, it can sense the surrounding situation in real time through sensors such as radar. When it detects abnormal situations such as someone approaching, the information will be notified to the owner in real time. The owner can use a mobile phone or other device to remotely view the surrounding situation of the vehicle in real time through the on-board camera. At this time, the image pushed to the owner needs to be desensitized. If the method of coding or smearing the rectangular frame in related technologies is used, the quality of the desensitized image will be rough.
[0061] By using the method proposed in this application, human faces can be replaced with styles such as cartoon faces, which increases the aesthetics and fun of real-time videos and helps improve the competitiveness of products.
[0062] In addition to the above-mentioned smart traffic scenarios, this application can also be applied to product applications such as remote automatic parking. These applications also require real-time video push to help car owners understand the situation around the vehicle in real time. The desensitization method proposed in this application can also achieve similar effects as above. And the application scenarios of this application are not limited to smart traffic scenarios, and can be used in any other scenarios that require face desensitization, such as security.
[0063] The image desensitization method provided in the embodiment of the present application can be implemented by a computer device, which can be a terminal device or a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, and this application is not limited here. The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.
[0064] The embodiments of the present application relate to artificial intelligence (AI), which is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.
[0065] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.
[0066] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0067] The embodiments of this application mainly involve the above-mentioned computer vision technology, machine learning, smart transportation and other major directions.
[0068] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, automatic driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0069] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.
[0070] For example, the embodiments of the present application can determine the recognition of the content to be desensitized in the image to be desensitized, the recognition of the outer contour of the content to be desensitized, etc. through computer vision technology, and can also determine the semantic similarity between the image sample and the reconstructed desensitized content, the semantic similarity between the desensitized sample and the reconstructed desensitized content, etc.
[0071] The embodiments of the present application can also train the content generation model and the style transfer model through machine learning.
[0072] It can be understood that in the specific implementation of the present application, before image desensitization is performed, the images to be desensitized and the image samples used for training will involve user faces waiting for desensitization. When the above embodiments of the present application are applied to specific products or technologies, each item needs to obtain user permission or consent separately, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0073] Figure 1A schematic diagram of an image desensitization scenario provided in an embodiment of the present application is provided. For the aforementioned computer device, the server 100 is exemplarily used as an example for explanation in this embodiment. The image to be desensitized including the content to be desensitized is exemplarily an image to be desensitized including the content to be desensitized of a real face.
[0074] When the server 100 obtains an image to be desensitized that includes real facial content, in order to protect the security of the content to be desensitized, the image to be desensitized can be desensitized. The server 100 determines the outer contour curve of the outer contour of the real facial content in the image to be desensitized, and the outer contour curve can accurately identify the area where the real facial content is located, and identify the shape of the real facial content in the image to be desensitized, thereby accurately distinguishing the boundary between the content to be desensitized and the non-content to be desensitized in the image to be desensitized.
[0075] Moreover, the outer contour curve can effectively identify the general structure of the content to be desensitized, such as the angle of the real face content, the size and width of the face, etc., which can be used as the basis for the server 100 to generate desensitized content. Figure 1 In the scene shown, the outer contour curve can be used to identify the real face content, which is a side face, a small face, etc. In order to avoid the leakage of possible privacy information in the attached figure, Figure 1 In the figure, a mosaic is added to avoid privacy leakage for the real face content. The mosaic is not the actual content of the image to be desensitized, that is, Figure 1 The image to be desensitized does not actually have this mosaic.
[0076] Thus, the server 100 can generate corresponding desensitized content according to the outer contour curve, and replace the desensitized content in the image to be desensitized with the desensitized content, without damaging the image quality of the non-desensitized content in the image to be desensitized. For example, the server 100 can use the content generation model 200 deployed by itself to determine the desensitized content corresponding to the outer contour curve.
[0077] Moreover, the desensitized content and the real face content to be desensitized have the same content type and different image styles, that is, the desensitized content is also a face type, and its content is also a face, but the image style is different, that is, it does not belong to the real image style, for example, it can belong to the animation style.
[0078] When the same content type makes the desensitized content in the image position of the original content to be desensitized, it does not stand out from the non-desensitized content in the image to be desensitized. For example Figure 1 There is hair around the real face content and the body part below. Even if the real face content is replaced with animated face content, a content association can be established between the surrounding hair and the body part below, and it will not appear abrupt.
[0079] Moreover, different image styles can reflect the difference in style between desensitized content and content not to be desensitized, effectively reminding users that this is desensitized content, not the original content in the image to be desensitized, to avoid misleading.
[0080] It can be seen that the image desensitization method provided by this application only replaces the content to be desensitized, and does not affect the image quality of the non-desensitized content part (for example, Figure 1 The image quality of the hair and body parts in the desensitized image is not affected), and a reasonable content connection is established between the replaced desensitized content and other non-desensitized content in the original desensitized image, and it will not appear abrupt. Therefore, the quality of the desensitized image is comprehensively improved from many aspects, and different image styles can bring fun to desensitization and improve the user experience.
[0081] Figure 2 A method flow chart of an image desensitization method provided in an embodiment of the present application, the method comprising:
[0082] S201: Acquire an image to be desensitized, wherein the image to be desensitized includes content to be desensitized.
[0083] The image to be desensitized is an image that includes the content to be desensitized, wherein the content to be desensitized can be any sensitive content or private content involving user privacy, such as the user's biological characteristics, sociological characteristics, etc. The biological characteristics can include face, fingerprints, palm prints, etc., and the sociological characteristics can include identity cards, certificate numbers, mobile phone numbers, etc. In addition, it can also include content that is harmful to the spread, such as improperly exposed limbs, improper behavior, etc., which is not limited in this application.
[0084] It should be noted that the image to be desensitized may include more than one content to be desensitized, for example, including multiple real faces. In this embodiment, one content to be desensitized is mainly used as an example for explanation. Different contents to be desensitized can be desensitized by reusing the image desensitization scheme of this application. For example, for an image to be desensitized that includes multiple contents to be desensitized, the image desensitization of the pair of desensitized images can be completed by implementing the image desensitization scheme of this application in parallel.
[0085] The present application can obtain images to be desensitized from a variety of sources, such as video frames acquired from acquisition equipment in the fields of smart transportation, security, etc., or historical images, etc.
[0086] In one possible implementation, the image to be desensitized is obtained by a collection device (such as a vehicle-mounted terminal) in the field of intelligent transportation. That is to say, the embodiment of the present application can perform real-time image desensitization tasks, and can also perform non-real-time image desensitization tasks. For example, when an external device requests to obtain a vehicle environment image, the vehicle-mounted terminal performs image desensitization processing on the vehicle environment image requested to be obtained. For example, the vehicle-mounted terminal can perform image desensitization on the collected vehicle environment image when system resources are relatively idle.
[0087] Image desensitization can prevent the user's desensitized content from being spread and viewed at will, thereby improving the security of image transmission in the relevant fields.
[0088] S202: Determine an outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized.
[0089] The outer contour of the content to be desensitized can accurately identify the boundary between the content to be desensitized and the content not to be desensitized in the image to be desensitized. The outer contour curve of the outer contour in the image to be desensitized identifies the exact range of the content to be desensitized in the image to be desensitized. Therefore, image desensitization based on the outer contour curve will not affect the normal display and image quality of the content not to be desensitized in the image to be desensitized.
[0090] This application does not limit how to determine the outer contour curve of the content to be desensitized. In a possible implementation, S202 includes:
[0091] S2021: Perform desensitization content detection on the image to be desensitized, and obtain a detection frame corresponding to the content to be desensitized in the image to be desensitized.
[0092] S2022: Generate an expanded cropped image corresponding to the content to be desensitized based on the detection frame and the image to be desensitized.
[0093] S2023: Segment the content to be desensitized on the expanded cropped image to obtain an outer contour curve corresponding to the outer contour of the content to be desensitized.
[0094] The above detection frame can identify the possible area of the to-be-desensitized content in the to-be-desensitized image, such as Figure 3 As shown in FIG. 1 , the image to be desensitized includes real face content as the content to be desensitized, wherein the corresponding face detection frame can be as shown in FIG. Figure 3 As shown in part (2) of the figure, in order to avoid the disclosure of possible privacy information, Figure 3 In the figure, a mosaic is added to the real face content to avoid privacy leakage. The mosaic is not the actual content of the image to be desensitized. For example, the image to be desensitized shown in parts (1) to (4) does not actually have the mosaic. Therefore, it is necessary to desensitize the content to be desensitized to avoid the real face from being improperly spread.
[0095] In order to improve the extraction accuracy of the outer contour curve and avoid incorrect extraction of the outer contour curve when there are multiple contents to be desensitized in the desensitized image, it is necessary to expand and crop the identified contents to be desensitized before extracting the outer contour curve to obtain an expanded cropped image, such as Figure 3 as shown in part (3).
[0096] The server can determine the outer contour curve corresponding to the outer contour of the content to be desensitized from the expanded cropped image by means of face segmentation, for example Figure 3 As shown in parts (4) and (5), the server determines a face mask of real face content, extracts an outer contour based on the face mask, and obtains a corresponding outer contour curve, as shown in part (5).
[0097] Through the above method, the extraction accuracy of the outer contour curve can be improved, and when there are multiple contents to be desensitized in the image to be desensitized, the wrong extraction of the outer contour curve can be avoided, such as extracting multiple contents to be desensitized as one content to be desensitized, while some non-content to be desensitized is included in the extracted outer contour curve. This improves the subsequent desensitization accuracy.
[0098] S203: Generate desensitized content whose shape conforms to the outer contour curve according to the outer contour curve.
[0099] The outer contour curve can accurately identify the area where the content to be desensitized is located and the shape of the area, thereby accurately distinguishing the boundary between the content to be desensitized and the non-content to be desensitized in the image to be desensitized. Moreover, the shape information reflected by the outer contour curve can identify the approximate structure of the content to be desensitized, so the server can accurately generate desensitized content whose shape conforms to the outer contour curve based on the outer contour curve, thereby reducing the possibility of covering or destroying the non-content to be desensitized in the image to be desensitized when the desensitized content replaces the content to be desensitized.
[0100] It should be noted that the desensitized content and the content to be desensitized have the same content type but different image styles.
[0101] The same content type makes the desensitized content in the image position of the original content to be desensitized not stand out from the non-desensitized content in the image to be desensitized, and the desensitized content can establish the original content association with the non-desensitized content. For example, when the original position of the real face content is replaced by the animated face content,
[0102] Since the content before and after the replacement belongs to the same content type, the replaced desensitized image will not lose the association between the desensitized content and the non-desensitized content. For example, compared with the abrupt situation of mosaic painting the desensitized content in the related art, replacing the desensitized content with the desensitized content of the same content type will make the desensitized content more integrated into the original image, and have less impact on the aesthetics of the image.
[0103] Different image styles can reflect the difference in style between desensitized content and content not to be desensitized, effectively reminding users that this is desensitized content, not the original content in the image to be desensitized, to avoid misleading.
[0104] In a possible implementation manner, the content type of the desensitized content and the content to be desensitized is face type, the image style of the desensitized content is cartoon style, and the image style of the content to be desensitized is reality style.
[0105] For example, when the original real face content is replaced with animated face content, since both belong to the human face type, the replaced desensitized image will not lose the connection between the animated face content and the non-desensitized content. For example, compared with the abrupt situation of mosaic painting of real faces in related technologies, using animated face content to replace real face content will make the animated face content more integrated into the original image, and have less impact on the aesthetics of the image.
[0106] As mentioned above, if a rectangular detection frame is used to code or smear a real face, the aesthetics of the image will be greatly reduced, affecting the user experience.
[0107] In this implementation, the real face content can be replaced with animated face content to increase the aesthetics of the desensitized image, especially when the image to be desensitized is a video frame in a real-time video. After desensitization, the user can see the animated face in the real-time video, which adds fun to the desensitization, improves the user experience, and is conducive to improving the competitiveness of image desensitization products.
[0108] S204: Obtain a desensitized image by replacing the content to be desensitized in the image to be desensitized with the desensitized content.
[0109] Since the shape of the desensitized content conforms to the outer contour curve of the content to be desensitized, after the image is desensitized, the desensitized content in the desensitized image will not affect other content not to be desensitized. While eliminating the content to be desensitized, it greatly avoids the damage to the content not to be desensitized in the image, and at the same time increases the aesthetics and interest of the image, which is conducive to the return use of image data or the development of downstream applications after desensitization.
[0110] It can be seen that when obtaining an image to be desensitized including content to be desensitized, in order to protect the safety of the content to be desensitized, the image to be desensitized can be subjected to image desensitization. The outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized can be determined, and the desensitized content whose shape conforms to the outer contour curve is generated. Since the outer contour curve can accurately distinguish the boundary between the content to be desensitized and the non-content to be desensitized in the image to be desensitized, the image quality of the non-content to be desensitized in the image to be desensitized will not be destroyed by replacing the content to be desensitized in the image to be desensitized with the desensitized content. Moreover, the desensitized content and the content to be desensitized have the same content type and different image styles. When the same content type makes the desensitized content in the image position of the original content to be desensitized, the non-content to be desensitized in the image to be desensitized is not abrupt in the image as a whole, and the desensitized content can establish the original content association between the content to be desensitized and the content to be desensitized with the non-content to be desensitized content. Different image styles can reflect the difference in style between the desensitized content and the non-content to be desensitized content, effectively prompting the user that this is the desensitized content, rather than the original content in the image to be desensitized, to avoid misleading. Therefore, the desensitized content generated by the outer contour curve of the content to be desensitized can effectively realize image desensitization and obtain a desensitized image that does not affect the image quality. Moreover, different image styles can bring fun to desensitization and improve the user experience.
[0111] As mentioned above, the server can generate desensitized content through a content generation model.
[0112] Therefore, in a possible implementation, S203 includes: generating, according to the outer contour curve, desensitized content having a shape conforming to the outer contour curve through a content generation model.
[0113] The content generation model is a model that generates desensitized content with the same content type as the content to be desensitized based on an outer contour curve, and the generated desensitized content and the content to be desensitized have different image styles. Therefore, for this purpose, an embodiment of the present application provides an optional method for training a content generation model, including the following steps. It should be noted that the training content generation model can be the aforementioned computer device or other devices, and the present application does not limit this.
[0114] S301: Obtain a training data set including contour-desensitized sample pairs.
[0115] The contour sample and the desensitized sample in the contour-desensitized sample pair have the same sample outer contour curve, and the desensitized sample and the desensitized content have the same image style.
[0116] That is to say, in order to train a content generation model that can generate desensitized content, it is necessary to use desensitized samples with the same image style as the desensitized content. For example, when the desensitized content is in anime style, the desensitized sample is also in anime style. Moreover, the desensitized sample is determined based on the desensitized content to be desensitized, that is, the content type of the desensitized sample is the same as the content type of the content to be desensitized.
[0117] In an embodiment of the present application, in each contour-desensitized sample pair in the training data set, the shape of the desensitized sample conforms to the sample outer contour curve identified by the contour sample in the contour-desensitized sample pair. This can correctly guide the content generation model to learn the relevant knowledge of generating desensitized content that conforms to the outer contour curve during the training process.
[0118] like Figure 4 Three contour-desensitized sample pairs in the training data set are shown, namely contour-desensitized sample pair 1, contour-desensitized sample pair 2 and contour-desensitized sample pair 3.
[0119] In each contour-mass-sensitized sample pair, the left side is the contour sample and the right side is the masks sample.
[0120] S302: According to the contour sample, obtain, through an initial generator, first predicted desensitized content having the same image style as the desensitized content.
[0121] S303: Obtain a first discrimination result and a second discrimination result through an initial discriminator according to the contour sample, the desensitized sample and the first predicted desensitized content.
[0122] The first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style.
[0123] S304: Generate a first adversarial loss function based on the first discrimination result and the second discrimination result, and perform adversarial training on the initial generator and the initial discriminator through the first adversarial loss function.
[0124] S305: According to the initial generator, the content generation model is obtained through the adversarial training.
[0125] Among them, the network structure composed of the initial generator and the initial discriminator can be as follows Figure 5 As shown, G is the initial generator and D is the initial discriminator. real is the contour sample in the contour-desensitized sample pair, B realis the desensitized sample in the contour-desensitized sample pair, B fake is the first predicted desensitized content. The initial discriminator is based on B real +A real Get the first judgment result, according to B f a ke +A real The second discrimination result is obtained.
[0126] The server may determine a first adversarial loss function L based on the first discrimination result and the second discrimination result. GAN (G, D), and based on the first adversarial loss function Figure 5 The network structure shown is subjected to adversarial training to train the initial generator D as the aforementioned content generation model.
[0127] The first adversarial loss function can be simplified as follows:
[0128] L GAN (G,D)=E x,y [logD(x,y)]+E x,z [log(1-D(x,G(x)))]
[0129] Among them, x is the contour sample, y is the desensitized sample, z is the first predicted desensitized content generated by x by the initial generator, and E is the expectation.
[0130] The essence of adversarial training refers to the adversarial relationship between the initial generator and the initial discriminator. That is, in adversarial training, the optimization goal is to make the first predicted desensitized content output by the initial generator deceive the initial discriminator, so that the initial discriminator determines that the first predicted desensitized content is a desensitized sample, and make the initial discriminator as accurate as possible, so that it can accurately identify whether the input is the first predicted desensitized content or the desensitized sample, and avoid being deceived by the initial generator.
[0131] Since the first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style, the first adversarial loss function determined by the first discrimination result and the second discrimination result can accurately guide how to generate the first predicted desensitized content that is more in line with the target type and the sample outer contour curve, and how to discriminate the first predicted desensitized content, so that the content generation model obtained by training should not only be similar to the desensitized content in image style, but also conform to the distribution of the content to be desensitized in the image sample in terms of content structure. Therefore, desensitized content that conforms to the target type and outer contour curve can be generated based on the input outer contour curve.
[0132] During the adversarial training, in one possible implementation, S304 includes:
[0133] By fixing the model parameters of the initial discriminator, the model parameters of the initial generator are adjusted with minimizing the adversarial loss function as the optimization goal;
[0134] By fixing the model parameters of the initial generator and taking maximizing the adversarial loss function as the optimization goal, the model parameters of the initial discriminator are adjusted.
[0135] That is, when adjusting the model parameters of the initial discriminator, the model parameters of the initial generator are not changed, and when adjusting the model parameters of the initial generator, the model parameters of the initial discriminator are not changed. The above training method of fixing one side and adjusting the other side can be performed alternately.
[0136] That is, the training method of the maximum and minimum zero-sum game is adopted, and the value function G * for:
[0137] G * = arg min G max D L GAN (G,D)
[0138] Through the above adversarial training method, the initial generator can be effectively adjusted to improve the model accuracy of the content generation model.
[0139] It should be noted that when obtaining the training data set, in general, the outer contour curves corresponding to the content to be desensitized are pre-labeled. For example, when the content to be desensitized is real face content, the collected outer contour curves of this type of content to be desensitized have been labeled, while the outer contour curves of the desensitized content generally do not have sufficient labeled data. If additional labeling is required, a lot of manpower and material resources will be wasted. Moreover, when the desensitized content is in anime style, there is still a difference between the contour of the anime face and the contour of the real face, which will cause the generated anime face to not fit the real face as the content to be desensitized sufficiently, making the desensitized content in the desensitized image not fit well with other non-desensitized content.
[0140] Therefore, for image samples including content to be desensitized, an embodiment of the present application provides an outer contour curve of the content to be desensitized that has been marked in the image sample as a sample outer contour curve. The image samples and the image styles of the images to be desensitized mentioned here are both source styles. This can greatly reduce the annotation cost and improve the training efficiency and training accuracy. Moreover, since the content generation model generates desensitized content based on the outer contour curve, although the generated desensitized content has the same content type as the content to be desensitized, it is not completely similar in content distribution. Taking a human face as an example, although the desensitized content is also a human face, its facial features, expressions, etc. will be different from the real human face in the content to be desensitized, which effectively reduces the degree of sensitive information in the desensitized content and achieves the desensitization effect with high quality.
[0141] In order to accurately obtain the matching sample outer contour curve and the desensitized sample to form a contour-desensitized sample pair, in a possible implementation, the training data set is obtained through a style transfer model, including the following steps:
[0142] S401: Acquire the image sample, wherein the outer contour of the content to be desensitized in the image sample is marked with the sample outer contour curve.
[0143] S402: According to the image sample, obtain the corresponding desensitized sample through a style conversion model.
[0144] S403: Construct the training data set based on the sample outer contour curve corresponding to the image sample and the desensitized sample.
[0145] Since the image sample used as the input of the style transfer model includes the content to be desensitized and the outer contour of the content to be desensitized has been marked, the style transfer model can accurately generate a desensitized sample whose shape conforms to the outer contour curve of the sample based on the guidance of the content distribution of the content to be desensitized in the image sample and the guidance of the outer contour curve of the sample in terms of range and shape, and the content distribution of the desensitized sample is close to the content distribution of the content to be desensitized in the image sample. Taking the content to be desensitized as real face content and the desensitized sample as anime face content as an example, the close content distribution may refer to the close distribution of faces in real face content and anime face content, for example, the position of the ears in the real face content is close to the position of the ears in the anime face content.
[0146] The style transfer model is a model that generates desensitized content with the same content type as the content to be desensitized based on an outer contour curve and the content to be desensitized. Moreover, the generated desensitized content and the content to be desensitized have different image styles. Therefore, an embodiment of the present application provides an optional method for training a style transfer model, including the following steps. It should be noted that the training content generation model can be the aforementioned computer device or other devices, and the present application does not limit this.
[0147] The style transfer model is trained in the following way:
[0148] S501: Obtain a desensitized image whose image style is the target style.
[0149] S502: According to the image sample, obtain a second predicted desensitized content through an initial source-target generator; and according to the desensitized image, obtain a predicted content to be desensitized through an initial target-source generator.
[0150] S503: obtaining a third discrimination result through an initial target style discriminator according to the second predicted desensitized content and the desensitized image, and obtaining a fourth discrimination result through an initial source style discriminator according to the predicted content to be desensitized and the image sample.
[0151] The third discrimination result is used to respectively identify whether the second predicted desensitized content and the desensitized image conform to the target style; the fourth discrimination result is used to respectively identify whether the first predicted content to be desensitized and the image sample conform to the source style.
[0152] S504: Generate an adversarial loss based on the third discrimination result and the fourth discrimination result.
[0153] S505: According to the second predicted desensitized content, obtain the reconstructed desensitized content through the initial target-source generator; and according to the predicted desensitized content, obtain the reconstructed desensitized content through the initial source-target generator.
[0154] S506: Generate a first reconstruction difference loss for identifying the difference between the image sample and the reconstructed desensitized content, and generate a second reconstruction difference loss for identifying the difference between the desensitized content and the reconstructed desensitized content.
[0155] S507: Generate a second adversarial loss function according to the adversarial loss, the first reconstruction difference loss, and the second reconstruction difference loss, and perform adversarial training on the initial source-target generator, the initial target-source generator, the initial target style discriminator, and the initial source style discriminator through the second adversarial loss function.
[0156] S508: According to the initial source-target generator, the style transfer model is obtained through the adversarial training.
[0157] The network structure of the style transfer model trained can be as follows Figure 6 As shown,
[0158] Among them G B is the initial source-target generator, G AInitial target-source generator, D B is the initial target style discriminator, D A Initial source style discriminator. A real is the image sample, B real For desensitized images, A fake To predict the content to be desensitized, B fake The second predicted desensitized content. rec To reconstruct the desensitized content, B rec To reconstruct the desensitized content. B It is used to finally generate the style transfer model. The dotted line part is used to indicate the process direction of generating and reconstructing the desensitized content and reconstructing the desensitized content.
[0159] Among them, the content type of the content to be desensitized is the source type, and the content type of the desensitized content is the target type. In this network structure, a two-way style transfer structure is constructed. One is the conversion and discrimination from the source type to the target type, which is performed by G B and D B The first step is the conversion and discrimination from the target type to the source type, which is composed of G A and D A The purpose of designing the network structure in this way is to reconstruct the predicted content generated by the generator (such as the predicted content to be desensitized and the second predicted desensitized content), so that the prediction ability of the generator can be tuned by the reconstruction difference loss obtained based on the reconstruction.
[0160] Taking the first reconstruction difference loss as an example, since the reconstructed content to be desensitized is obtained based on the second predicted desensitized content generated by the initial source-target generator, the quality of the second predicted desensitized content in content distribution can be reflected in the difference between the reconstructed content to be desensitized and the image sample.
[0161] The difference is mainly reflected in the image semantics. For example, if the ears of the reconstructed desensitized content are consistent with the positions of the ears in the image sample, the difference in the ears is relatively small. If the nose of the reconstructed desensitized content is inconsistent with the positions of the nose in the image sample, the difference in the nose is relatively large. The semantic distribution in the reconstructed desensitized content is mainly determined by the aforementioned second predicted desensitized content. That is to say, if the second predicted desensitized content is closer to the semantic distribution of the image sample, the difference between the reconstructed desensitized content and the image sample based on the second predicted desensitized content is smaller.
[0162] For each style transfer structure in the above network structure, adversarial loss can also be generated through adversarial training ideas similar to those used in the aforementioned training content generation model.
[0163] Taking a style transfer structure consisting of an initial source-target generator and an initial target style discriminator as an example, the optimization goal of the constructed adversarial loss is to allow the second predicted desensitized content output by the initial source-target generator to deceive the initial target style discriminator, so that the initial target style discriminator determines that the first predicted desensitized content is a desensitized sample, and make the initial target style discriminator as accurate as possible, so that it can accurately identify whether the input is the first predicted desensitized content or the desensitized sample, so as to avoid being deceived by the initial source-target generator.
[0164] The aforementioned expression of adversarial loss can be broken down into two, corresponding to different style transfer structures.
[0165] For a style transfer structure consisting of an initial source-target generator and an initial target style discriminator, it can be expressed as follows:
[0166]
[0167] Among them, p data (B real ) is used to identify B real The extraction range can be understood as a collection of desensitized images; data (A real ) is used to identify A real The extraction range can be understood as a set of image samples.
[0168] For a style transfer structure consisting of an initial target-source generator and an initial source style discriminator, it can be expressed as follows:
[0169]
[0170] The expression of the first reconstruction difference loss can be expressed as follows:
[0171]
[0172] The expression of the second reconstruction difference loss can be expressed as follows:
[0173]
[0174] The final expression of the second adversarial loss function can be expressed as follows:
[0175]
[0176] Among them, λ is the weight.
[0177] Since the third discrimination result is used to respectively identify whether the second predicted desensitized content and the desensitized image conform to the target style; the fourth discrimination result is used to respectively identify whether the first predicted content to be desensitized and the image sample conform to the source style. Therefore, the adversarial loss determined by the third discrimination result and the fourth discrimination result can accurately guide how to generate the second predicted desensitized content that is more in line with the target type, and how to discriminate the second predicted desensitized content. Since the first reconstruction difference loss is used to identify the difference between the image sample and the reconstructed content to be desensitized, the second reconstruction difference loss is used to identify the difference between the desensitized content and the reconstructed desensitized content. Therefore, under the guidance of the second adversarial loss function generated according to the adversarial loss, the first reconstruction difference loss and the second reconstruction difference loss, the style transfer model can also generate a second predicted desensitized content with a content distribution similar to that of the image sample.
[0178] During the adversarial training, in one possible implementation, S507 includes:
[0179] By fixing the model parameters of the initial target style discriminator and the initial source style discriminator, the model parameters of the initial source-target generator and the initial target-source generator are adjusted with minimizing the second adversarial loss function as the optimization goal.
[0180] By fixing the model parameters of the initial source-target generator and the initial target-source generator, the model parameters of the initial target style discriminator and the initial source style discriminator are adjusted with maximizing the second adversarial loss function as the optimization goal.
[0181] The above training method of fixing one side and adjusting the other side can be performed alternately.
[0182] That is, the training method of the maximum and minimum zero-sum game is adopted, and the value function G * for:
[0183] G * = arg min G max D Loss
[0184] Through the above adversarial training method, the initial source-target generator can be effectively adjusted to improve the model accuracy of the style transfer model.
[0185] For the initial source-target generator and the initial target-source generator as generators, the generator network structure of any one of them can be as follows Figure 7 As shown, a context information extraction module is provided between the encoder and the decoder.
[0186] The context information extraction module includes dilated convolutional layers with different dilation rates, and the dilated convolutional layers with different dilation rates are used to improve the receptive field of the decoder.
[0187] When designing the generator network, the network input is a red, green, and blue RGB image (such as an image sample), and the output is a three-channel feature (desensitized sample) with the same resolution as the input.
[0188] Based on desensitization considerations, in order to avoid retaining too much detail information, in the desensitization process of style transfer, the length and width of the feature are Figure 7 The downsampling module in is downsampled to 1 / 4 of the original. Since the downsampling multiple is too small, it is easy to cause insufficient context information, which will result in less information being obtained by the subsequent decoder.
[0189] To this end, a context information extraction module is set between the encoder and the decoder. The structure of the context information extraction module is a dilated convolutional pyramid composed of different dilation rates, so as to increase the network receptive field for the subsequent decoder. Figure 7 Four different expansion rates are shown in the figure. Rectangular boxes of different sizes in each rectangular feature map are used to represent the corresponding receptive fields after the extended convolution with different expansion rates.
[0190] The advantage of using multiple expansion rates is that the context information output by multiple expansion rates can be combined to increase the probability of providing an effective receptive field for the decoder, which helps to improve the restoration accuracy of the decoder.
[0191] Finally, a lighter decoder is used to restore the features to the input resolution.
[0192] The network structures of the initial source-target generator and the initial target-source generator can be as follows Figure 7 As shown. For example, it can be responsible for the style conversion from real face content to cartoon face content, and from cartoon face content to real face content.
[0193] exist Figure 7 The possible structures of the residual convolution module, downsampling module and upsampling module included in the network structure are also shown.
[0194] Among them, the residual convolution module includes a convolution layer (conv), an instance norm layer (InstanceNorm, a function used for stylized migration), and the instance norm layer also includes ReLU (an activation function) processing.
[0195] The downsampling module includes two parallel inputs before the concatenation layer, namely the maximum pooling layer and the convolution layer, and the stride used is 2.
[0196] The upsampling module is equipped with a transpose layer before the instance normalization layer, and the stride adopted is 2.
[0197] The initial target style discriminator and the initial source style discriminator included in the network structure of the style transfer model obtained by the above training are both discriminators. The input of the discriminator is the second predicted desensitized content or desensitized image output by the corresponding generator, and the output is a feature map with both length and width downsampled to 1 / 16 of the original. The number of feature map channels is 1. Its structure can be as follows Figure 8 As shown, Figure 8 The right side of shows the output feature map size of the corresponding layer.
[0198] You can use two independent Figure 8 The discriminators shown correspond to the initial target style discriminator and the initial source style discriminator, respectively.
[0199] Since both the content generation model and the style transfer model convert a source type input image into a target type output image, in one possible implementation, the initial generator, the initial source-target generator, and the initial target-source generator have the same model structure, and the initial discriminator, the initial target style discriminator, and the initial source style discriminator have the same model structure.
[0200] Since the style transfer model is used to determine the training data set, it is generally completed before the content generation model is built. Therefore, when building a network structure for the content generation model, for the initial generator in the network structure, the model structure of the initial source-target generator can be reused to obtain the initial generator, and for the initial discriminator, the model structure of the initial target style discriminator can be reused to obtain the initial discriminator. This improves the efficiency of building the model structure in the embodiment of the present application.
[0201] For the entire training process of the style transfer model and the training data set generated based on the style transfer model, the desensitized images, image samples, sample outer contour curves and desensitized samples involved can be as follows: Fig. 9 As shown in the figure, it can be seen that the desensitized samples generated based on the image samples and the identified sample outer contour curves are not only highly distinguishable in content type, but also similar in distribution to the image samples in terms of image semantics, which can effectively guide the subsequent training of content generation models. Desensitized images do not need to have content association with image samples, as long as they meet the target type, they can be used as desensitized images.
[0202] It should be noted that in Fig. 9 In order to avoid leaking possible privacy information in the attached image, Fig. 9 In the figure, a mosaic is added to the real face content of the image sample to avoid privacy leakage. The mosaic is not the actual content of the image sample, that is, there is no mosaic on the actual image sample.
[0203] Next, based on the image desensitization scenario, combined with Fig.10 and Fig.11 The embodiments of the present application are described as a whole. Fig.10 and Fig.11 In the scene shown, the content to be desensitized is real face content, and the desensitized content is animated face content.
[0204] against Fig.11 , and the aforementioned Figure 3 and Fig. 9 For the same reason, in order to avoid the leakage of possible privacy information in the attached pictures, Fig.11 In the figure, a mosaic is added to the real face content of the image sample to avoid privacy leakage. The mosaic is not the actual content of the image sample, that is, there is no mosaic on the actual image sample.
[0205] S11: Face detection
[0206] For the images to be desensitized, such as Fig.11 As shown in part (1) of , face detection can be performed using a face detection model to determine a face detection frame, such as Fig.11 as shown in part (2).
[0207] S12: Face expansion and cropping
[0208] Based on the face detection frame, the face image can be obtained by expansion and cropping, such as Fig.11 as shown in part (3).
[0209] S13: Segmentation of face mask
[0210] The face segmentation model can be used to segment the face image and extract the face mask from the face image, such as Fig.11 as shown in part (4).
[0211] S14: Extract outer contour curve
[0212] Since the face mask can identify the outer contour of the real face, the outer contour curve of the real face content in the image to be desensitized can be extracted based on the face mask, such as Fig.11 as shown in part (5).
[0213] S15: Generate anime face based on outer contour curve
[0214] Based on the above content, a model can be generated to determine the anime face content corresponding to the outer contour curve, such as Fig.11 The shape of the cartoon face content conforms to the outer contour curve extracted in the previous step, and the distribution of facial features is similar to that of the real face content.
[0215] S16: Replace real faces with animated faces
[0216] Since the embodiments of the present application can generate high-quality animated facial content, the desensitized image obtained by replacing the real facial content in the image to be desensitized is not abrupt in the overall image, and the desensitized content can establish the original content association between the content to be desensitized and the content not to be desensitized. Different image styles can reflect the difference in painting style between the desensitized content and the content not to be desensitized, effectively prompting the user that this is the desensitized content, not the original content in the image to be desensitized, to avoid misleading, and different image styles can bring fun to desensitization and improve user experience.
[0217] The desensitized image with real face content replaced by S16 can be Fig.11 as shown in part (7).
[0218] Next, the detection model for determining the detection frame and the segmentation model for segmenting the content to be desensitized are described in combination with S2021 and S2023. The face detection model for determining the face detection frame mentioned in the above embodiment can be a detection model, and the face segmentation model for determining the face mask mentioned in the above embodiment can be a segmentation model.
[0219] With respect to the aforementioned S2021: performing desensitization content detection on the image to be desensitized, and obtaining a detection frame corresponding to the content to be desensitized in the image to be desensitized.
[0220] The detection frame of the content to be desensitized can be determined by the detection model. The training and design ideas of the detection model are as follows (taking the content to be desensitized as real face content as an example):
[0221] Determine the number of anchor boxes B at each position of the network output layer, consider the width and height of the face rectangle box as features, and use k-mean to cluster all face rectangle boxes into B categories, and then take the centroid of this B category as the width and height of the corresponding anchor box.
[0222] Design a detection model for faces. The network structure can be as follows Fig.12 As shown in the figure, since the face is relatively small, the network only downsamples to 1 / 8 of the original size to maintain high-resolution features. The number of channels in the output layer is B*(4+1+c), where B is the number of anchor boxes, 4 represents the offset regression of the center horizontal coordinate, center vertical coordinate, length, and width of each anchor box, 1 represents the confidence of whether it is a target, and c is the target category, which is 1 here. Therefore, the output feature channel is B*6.
[0223] exist Fig.12In the figure, ConvBnRelu is a network structure of Conv (convolution) + BN (normalization, Batch Normalization) + Relu (rectified linear unit, Rectified Linear Unit).
[0224] Res Block is the basic module of ResNet.
[0225] The parameters on the right side of the dashed box are the size parameters of the output features of the corresponding network structure. Fig.12 The specific composition of the two network structures, ConvBnRelu and Res Block, is also shown.
[0226] For the training of the detection model, the loss function is determined as:
[0227]
[0228] The first and second lines are the offset regression losses of the predicted box relative to the center point and width and height of the anchor box. S represents the width and height of the output feature map. B represents the number of anchor boxes at each position of the output feature map mentioned above. It means asking whether the (i, j) position of the output feature map is on the target. If so, the value is 1, otherwise it is 0. The third line indicates whether there is a target confidence loss. The fourth line is the category loss, and the sum of the losses of each category is calculated on the output feature map. α, β, and γ represent the weights of the loss.
[0229] The detection model can be based on Fig. 9 The image samples are combined with the corresponding face detection frames for training.
[0230] With respect to the aforementioned S2023: segment the content to be desensitized on the expanded cropped image to obtain an outer contour curve corresponding to the outer contour of the content to be desensitized.
[0231] The segmentation model can be used to segment the content to be desensitized to extract the outer contour curve. The training and design ideas of the segmentation model are as follows (taking the content to be desensitized as real face content as an example):
[0232] Face segmentation can reuse the network structure in the initial generator (e.g. Figure 7 As shown in the figure), the downsampling ratio is adjusted to 1 / 8 and the number of output feature map channels is adjusted to 2.
[0233] The loss function for the segmentation model can use the cross entropy loss:
[0234]
[0235] Among them, c represents the number of categories, y cIt is a one-hot vector, whose elements can only take values of 0 and 1. Its length is the number of categories. Each position represents a category. If the category corresponding to a certain position is the marked true category, it takes 1, otherwise it takes 0. c Represents the probability that the predicted sample belongs to c.
[0236] The segmentation model can be based on Fig. 9 The image samples are combined with the corresponding annotated face masks for training.
[0237] The present application embodiment also provides an image desensitization method based on a vehicle-side scene. Fig.13 In this embodiment, the vehicle-mounted terminal is used as an example of the aforementioned computer device. The vehicle-mounted terminal can provide users with video and image acquisition services of the target vehicle's exterior environment, and can perform local desensitization work based on whether the collected video and image contain content to be desensitized, so as to avoid possible privacy leakage caused by the transmission of content to be desensitized. In addition, the vehicle-mounted terminal can also provide other intelligent service functions, such as voice control, application installation, video playback, vehicle performance management, etc. Based on Fig.13 , the above method comprises the following steps:
[0238] S1301: Acquire a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle.
[0239] The vehicle-mounted terminal can collect vehicle environment images of the target vehicle in different directions and angles. This application does not limit this. Based on the different directions and angles, the vehicle environment image can at least include an external vehicle environment image and an internal vehicle environment image. The external vehicle environment image is the vehicle environment image obtained by the vehicle-mounted terminal collecting the external environment of the target vehicle, and the internal vehicle environment image is the vehicle environment image obtained by the vehicle-mounted terminal collecting the internal environment of the target vehicle.
[0240] For example, the vehicle exterior environment image may include the environment images collected by the vehicle terminal from the front and rear of the target vehicle, as well as the environment images collected from both sides of the target vehicle, etc. The vehicle interior environment image may include the environment images collected by the vehicle terminal from inside the target vehicle, such as the environment images for the driver, the co-pilot, and the back row, etc.
[0241] The image collection can be performed during driving, for example, when the target vehicle is driving normally or during assisted driving, the vehicle terminal collects the vehicle environment image of the target vehicle. The vehicle terminal can also collect the image when preparing to park, or when the target vehicle is parked. In other words, when someone passes by the target vehicle (whether it is driving or parked) or enters the target vehicle, the vehicle environment image including the person may be collected by the vehicle terminal of the target vehicle.
[0242] For example, in the aforementioned parking sentry mode, radar and other sensors are used to perceive the surrounding situation in real time. When abnormal situations such as someone approaching or entering the car are detected, the corresponding vehicle environment image will be collected and notified to the owner after real-time desensitization. The owner can use personal devices such as mobile phones to call the on-board camera through the APP to remotely view the desensitized image in real time or view the historical desensitized images saved locally on the on-board terminal to understand the situation around or inside the vehicle.
[0243] In addition, in various smart transportation scenarios such as remote automatic parking, it is necessary to collect vehicle environment images through the on-board terminal, which will not be elaborated here.
[0244] It should be noted that the vehicle environment image obtained in this step can be a single static image or a frame in a dynamic video, that is, the vehicle environment image is a video frame in the vehicle environment video captured by the vehicle terminal. In other words, in this embodiment, a real-time desensitization service for detecting and desensitizing content to be desensitized can be implemented for static images or dynamic videos in the vehicle-side scenario.
[0245] S1302: When it is determined that the vehicle environment image includes content to be desensitized, generate corresponding desensitized content according to the content to be desensitized.
[0246] Since the target vehicles are mostly in human activity areas, such as driving and parking scenarios, there will be people passing by from time to time. Therefore, when the on-board terminal of the target vehicle collects the vehicle environment image, the vehicle environment image will often include human portrait content, especially real human face content. These are human biological characteristics and will be identified by the on-board terminal as content to be desensitized. Image desensitization is required to avoid the improper spread of the content to be desensitized.
[0247] Therefore, when the vehicle terminal determines that the vehicle environment image has content to be desensitized similar to the above-mentioned real face content, it will generate corresponding desensitized content according to the content to be desensitized, and the desensitized content has the same content type and different image style as the content to be desensitized. For the definition of the desensitized content, please refer to the description of the above-mentioned embodiment, which will not be repeated here.
[0248] In this embodiment, there is no limitation on how to obtain the corresponding desensitized content through the content to be desensitized. For example, in one possible implementation, S1302 includes: determining an outer contour curve of the outer contour of the content to be desensitized in the vehicle environment image; generating the desensitized content having a shape conforming to the outer contour curve according to the outer contour curve; and obtaining a desensitized environment image by replacing the content to be desensitized in the vehicle environment image with the desensitized content.
[0249] For the specific description of this implementation method, please refer to the relevant parts of the above-mentioned embodiment, which will not be repeated here. Figure 1-Figure 12 The method provided by the corresponding embodiment is implemented.
[0250] S1303: Obtain a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content.
[0251] Through image desensitization, the desensitized content in the vehicle environment image can be replaced with desensitized content without sensitive information. Even if the desensitized environment image is obtained by other external devices, the desensitized content cannot be obtained based on the desensitized environment image, such as restoring the real face content, thereby avoiding improper leakage of the desensitized content and protecting the security of the user's desensitized content.
[0252] It should be noted that, in order to protect the security of the user's content to be desensitized, the image desensitization of the vehicle environment image is completed on the vehicle terminal side, and the vehicle environment image with the content to be desensitized does not need to be sent to the server or other external device to achieve image desensitization. For example, through the lightweight content generation model provided in the aforementioned embodiment, real-time or non-real-time image desensitization of the image to be desensitized can be quickly achieved locally on the vehicle terminal.
[0253] Moreover, since the vehicle terminal performs image desensitization locally, it does not rely on networking. Even if the vehicle terminal is offline and not connected to the network, it will not affect the image desensitization work of the vehicle terminal. This makes the image desensitization method provided in the embodiment of the present application more applicable and more secure.
[0254] In order to further improve the security of the content to be desensitized, in a possible implementation, after obtaining the desensitized environment image corresponding to the vehicle environment image, the vehicle environment image can be deleted locally on the vehicle terminal. As a result, the vehicle environment image including the content to be desensitized cannot be obtained by either the vehicle-mounted device or the external device, which greatly increases the possibility of the content to be desensitized being leaked.
[0255] However, in some scenarios, such as parking scenarios, the vehicle environment image including the content to be desensitized may be related to some abnormal events targeting the target vehicle, such as when the target vehicle is scratched or maliciously damaged, etc. Therefore, the vehicle terminal can also selectively retain the vehicle environment image locally when identifying such situations. The user or relevant personnel of the target vehicle can view the vehicle environment image at the vehicle terminal to determine the cause of the abnormal behavior.
[0256] It should be noted that even though the vehicle environment images can be retained locally on the vehicle terminal in some cases, these vehicle environment images (including the content to be desensitized) can only be viewed through the vehicle terminal and cannot be transferred from the vehicle terminal to other external devices through various transmission methods, such as the user’s mobile phone, network disk, etc., thereby ensuring the security of the content to be desensitized while retaining the vehicle environment images.
[0257] S1304: In response to an acquisition request sent by an external device for the vehicle environment image, returning the desensitized environment image to the external device.
[0258] Since the vehicle terminal generally has a data connection with an external device such as a server or a user's personal terminal, the external device has the ability to request the vehicle environment image collected from the vehicle terminal. The external device is a device other than the vehicle terminal, and may include a mobile device such as a user's mobile phone, or a server corresponding to the vehicle terminal.
[0259] For example, the user of the target vehicle can request to retrieve the vehicle environment image of a certain period of time or the real-time vehicle environment image through the mobile phone APP. The server corresponding to the vehicle terminal can also request to obtain the vehicle environment image of the vehicle terminal, etc. In these cases, the vehicle terminal will receive the corresponding acquisition request.
[0260] When receiving such a request for obtaining a vehicle environment image, the vehicle-mounted terminal may return a desensitized environment image generated based on the vehicle environment image instead of the vehicle environment image.
[0261] The present application does not limit the timing when the vehicle-mounted terminal obtains the acquisition request sent by the external device. For example, the acquisition request can be actively initiated by the external device to the vehicle-mounted terminal, such as when the user wants to view the vehicle environment image last night and the server periodically obtains the vehicle environment image; it can also be passively initiated to the vehicle-mounted terminal under the prompt of the vehicle-mounted terminal. For example, when the vehicle-mounted terminal recognizes that an abnormal event has occurred for the target vehicle, it will send a prompt message to the external device such as the user's mobile phone of the target vehicle. The user can request to obtain the corresponding vehicle environment image under the prompt of the prompt message so that he can view it on the mobile phone in time.
[0262] Therefore, in a possible implementation manner, the method further includes:
[0263] When it is determined that an abnormal event occurs for the target vehicle, a prompt message is sent to the external device, wherein the vehicle environment image is collected by the vehicle-mounted terminal when the abnormal event occurs;
[0264] The acquisition request sent by the external device based on the prompt information is acquired.
[0265] Generally, the vehicle-mounted terminal will not actively send prompt information to related external devices if the target vehicle has not encountered an abnormal event. Users can choose whether to view the vehicle environment image based on their own viewing needs. When the target vehicle encounters an abnormal event, from a safety perspective, it is necessary for the user of the target vehicle to be able to detect it as early as possible. Therefore, when the vehicle-mounted terminal finds that the target vehicle has encountered an abnormal event, it will send a prompt message to the corresponding external device to prompt the target vehicle that it has encountered an abnormal event and is in an unsafe situation.
[0266] The vehicle-mounted terminal can collect vehicle environment images during the occurrence of abnormal events to record the specific circumstances of the abnormal events as much as possible. Therefore, when the user obtains the vehicle environment images based on prompt information through an external device, the vehicle-mounted terminal can return the desensitized environment image to the external device after the image is desensitized, so that the user can view the specific circumstances of the abnormal event.
[0267] It can be seen that in the scenario of smart transportation, the on-board terminal of the target vehicle can collect the vehicle environment image, and when it is determined that the vehicle environment image has content to be desensitized, generate desensitized content for replacement, and obtain the desensitized environment image corresponding to the vehicle environment image. When the external device obtains the acquisition request for the vehicle environment image, the desensitized environment image will be returned instead of the vehicle environment image, and the vehicle environment image with the content to be desensitized will not be transmitted, thereby effectively avoiding the improper leakage of the content to be desensitized in the smart transportation scenario, and greatly improving the security of the content to be desensitized in the smart transportation scenario.
[0268] In the aforementioned Figure 1-Figure 12 Based on the corresponding embodiment, Fig.14 The device structure diagram of an image desensitization device provided in an embodiment of the present application, the image desensitization device 1400 includes an acquisition unit 1401, a determination unit 1402, a generation unit 1403 and a replacement unit 1404:
[0269] The acquisition unit 1401 is used to acquire the image to be desensitized, wherein the image to be desensitized includes the content to be desensitized;
[0270] The determining unit 1402 is used to determine an outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized;
[0271] The generating unit 1403 is configured to generate, according to the outer contour curve, desensitized content having a shape conforming to the outer contour curve, wherein the desensitized content and the content to be desensitized have the same content type and a different image style;
[0272] The replacing unit 1404 is configured to obtain a desensitized image by replacing the content to be desensitized in the image to be desensitized with the desensitized content.
[0273] In a possible implementation manner, the generating unit is further configured to:
[0274] According to the outer contour curve, generating desensitized content having a shape conforming to the outer contour curve through a content generation model;
[0275] If the image style of the desensitized content is the target style, the device further includes a first training unit, and the first training unit is used to train the content generation model in the following manner:
[0276] Acquire a training data set including contour-desensitized sample pairs, wherein the contour sample and the desensitized sample in the contour-desensitized sample pair have the same sample outer contour curve, and the desensitized sample and the desensitized content have the same image style;
[0277] According to the contour sample, obtaining, by an initial generator, first predicted desensitized content having the same image style as the desensitized content;
[0278] According to the contour sample, the desensitized sample and the first predicted desensitized content, a first discrimination result and a second discrimination result are obtained by an initial discriminator, wherein the first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style;
[0279] Generate a first adversarial loss function based on the first discrimination result and the second discrimination result, and perform adversarial training on the initial generator and the initial discriminator through the first adversarial loss function;
[0280] According to the initial generator, the content generation model is obtained through the adversarial training.
[0281] In a possible implementation, the sample outer contour curve is determined according to the outer contour of the content to be desensitized in the image sample, and the image styles of the image sample and the image to be desensitized are both source styles.
[0282] In a possible implementation manner, the first training unit is further configured to:
[0283] By fixing the model parameters of the initial discriminator, the model parameters of the initial generator are adjusted with minimizing the adversarial loss function as the optimization goal;
[0284] By fixing the model parameters of the initial generator and taking maximizing the adversarial loss function as the optimization goal, the model parameters of the initial discriminator are adjusted.
[0285] In a possible implementation manner, the acquiring unit is further configured to:
[0286] Acquire the image sample, wherein the outer contour of the content to be desensitized in the image sample is marked with the sample outer contour curve;
[0287] According to the image sample, obtaining the corresponding desensitized sample through a style conversion model;
[0288] The training data set is constructed based on the sample outer contour curve corresponding to the image sample and the desensitized sample.
[0289] In a possible implementation, the device further includes a second training unit, and the second training unit obtains the style transfer model by training in the following manner:
[0290] Obtain a desensitized image whose image style is the target style;
[0291] According to the image sample, obtaining a second predicted desensitized content through an initial source-target generator; and according to the desensitized image, obtaining a predicted content to be desensitized through an initial target-source generator;
[0292] According to the second predicted desensitized content and the desensitized image, a third discrimination result is obtained by an initial target style discriminator, and the third discrimination result is used to respectively identify whether the second predicted desensitized content and the desensitized image conform to the target style; and according to the predicted content to be desensitized and the image sample, a fourth discrimination result is obtained by an initial source style discriminator, and the fourth discrimination result is used to respectively identify whether the first predicted content to be desensitized and the image sample conform to the source style;
[0293] generating an adversarial loss based on the third discrimination result and the fourth discrimination result;
[0294] According to the second predicted desensitized content, reconstructing the content to be desensitized through the initial target-source generator; and according to the predicted content to be desensitized, reconstructing the desensitized content through the initial source-target generator;
[0295] generating a first reconstruction difference loss for identifying a difference between the image sample and the reconstructed desensitized content, and generating a second reconstruction difference loss for identifying a difference between the desensitized content and the reconstructed desensitized content;
[0296] generating a second adversarial loss function according to the adversarial loss, the first reconstruction difference loss, and the second reconstruction difference loss, and performing adversarial training on the initial source-target generator, the initial target-source generator, the initial target style discriminator, and the initial source style discriminator through the second adversarial loss function;
[0297] According to the initial source-target generator, the style transfer model is obtained through the adversarial training.
[0298] In a possible implementation manner, the second training unit is further configured to:
[0299] By fixing the model parameters of the initial target style discriminator and the initial source style discriminator, the model parameters of the initial source-target generator and the initial target-source generator are adjusted with minimization of the second adversarial loss function as the optimization goal;
[0300] By fixing the model parameters of the initial source-target generator and the initial target-source generator, the model parameters of the initial target style discriminator and the initial source style discriminator are adjusted with maximizing the second adversarial loss function as the optimization goal.
[0301] In a possible implementation, a context information extraction module is arranged between the encoder and decoder of the initial source-target generator and the initial target-source generator, and the context information extraction module includes dilated convolution layers with different dilation rates, and the dilated convolution layers with different dilation rates are used to improve the receptive field of the decoder.
[0302] In a possible implementation, the initial generator, the initial source-target generator and the initial target-source generator have the same model structure, and the initial discriminator, the initial target style discriminator and the initial source style discriminator have the same model structure.
[0303] In a possible implementation manner, the determining unit is further configured to:
[0304] Performing desensitization content detection on the image to be desensitized, and obtaining a detection frame corresponding to the content to be desensitized in the image to be desensitized;
[0305] Generate an expanded cropped image corresponding to the content to be desensitized according to the detection frame and the image to be desensitized;
[0306] The expanded cropped image is segmented into the content to be desensitized to obtain an outer contour curve corresponding to the outer contour of the content to be desensitized.
[0307] In a possible implementation, the image to be desensitized is obtained through a collection device in the field of smart transportation.
[0308] In a possible implementation manner, the content type of the desensitized content and the content to be desensitized is face type, the image style of the desensitized content is cartoon style, and the image style of the content to be desensitized is reality style.
[0309] It can be seen that when obtaining an image to be desensitized including content to be desensitized, in order to protect the safety of the content to be desensitized, the image to be desensitized can be subjected to image desensitization. The outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized can be determined, and the desensitized content whose shape conforms to the outer contour curve is generated. Since the outer contour curve can accurately distinguish the boundary between the content to be desensitized and the non-content to be desensitized in the image to be desensitized, the image quality of the non-content to be desensitized in the image to be desensitized will not be destroyed by replacing the content to be desensitized in the image to be desensitized with the desensitized content. Moreover, the desensitized content and the content to be desensitized have the same content type and different image styles. When the same content type makes the desensitized content in the image position of the original content to be desensitized, the non-content to be desensitized in the image to be desensitized is not abrupt in the image as a whole, and the desensitized content can establish the original content association between the content to be desensitized and the content to be desensitized with the non-content to be desensitized content. Different image styles can reflect the difference in style between the desensitized content and the non-content to be desensitized content, effectively prompting the user that this is the desensitized content, rather than the original content in the image to be desensitized, to avoid misleading. Therefore, the desensitized content generated by the outer contour curve of the content to be desensitized can effectively realize image desensitization and obtain a desensitized image that does not affect the image quality. Moreover, different image styles can bring fun to desensitization and improve the user experience.
[0310] In the aforementioned Fig.13 Based on the corresponding embodiment, Fig.15 This is a device structure diagram of an image desensitization device provided in an embodiment of the present application. The image desensitization device 1500 includes an acquisition unit 1501, a generation unit 1502, a replacement unit 1503 and a sending unit 1504:
[0311] The acquisition unit 1501 is used to acquire a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle;
[0312] The generating unit 1502 is configured to generate corresponding desensitized content according to the content to be desensitized when it is determined that the vehicle environment image includes the content to be desensitized, wherein the desensitized content has the same content type and a different image style as the content to be desensitized;
[0313] The replacing unit 1503 is used to obtain a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content;
[0314] The sending unit 1504 is used to return the desensitized environment image to the external device in response to an acquisition request sent by the external device for the vehicle environment image.
[0315] In a possible implementation manner, the device further includes a deleting unit, and the deleting unit is used to delete the vehicle environment image in the vehicle terminal.
[0316] In a possible implementation, the vehicle environment image is a video frame in the vehicle exterior environment video captured by the vehicle-mounted terminal.
[0317] In a possible implementation, the sending unit is further configured to send prompt information to the external device when it is determined that an abnormal event occurs for the target vehicle, and the vehicle environment image is collected by the vehicle-mounted terminal when the abnormal event occurs;
[0318] The acquisition unit is further configured to acquire the acquisition request sent by the external device based on the prompt information.
[0319] It can be seen that in the scenario of smart transportation, the on-board terminal of the target vehicle can collect the vehicle environment image, and when it is determined that the vehicle environment image has content to be desensitized, generate desensitized content for replacement, and obtain the desensitized environment image corresponding to the vehicle environment image. When the external device obtains the acquisition request for the vehicle environment image, the desensitized environment image will be returned instead of the vehicle environment image, and the vehicle environment image with the content to be desensitized will not be transmitted, thereby effectively avoiding the improper leakage of the content to be desensitized in the smart transportation scenario, and greatly improving the security of the content to be desensitized in the smart transportation scenario.
[0320] The embodiment of the present application further provides a computer device, which is the computer device described above and may include a terminal device or a server, and the image desensitization device described above may be configured in the computer device. The computer device is described below in conjunction with the accompanying drawings.
[0321] If the computer device is a terminal device, see Fig.16 As shown, the embodiment of the present application provides a terminal device, taking the terminal device as a mobile phone as an example:
[0322] Fig.16 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.16 The mobile phone includes: a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art can understand that Fig.16 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0323] Combine the following Fig.16 A detailed introduction to the various components of the mobile phone:
[0324] The RF circuit 1410 may be used for receiving and sending signals during information transmission or communication or during a call. In particular, after receiving downlink information from a base station, the information is sent to the processor 1480 for processing. In addition, the uplink data is sent to the base station.
[0325] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1420. The memory 1420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0326] The input unit 1430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432 .
[0327] The display unit 1440 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1440 may include a display panel 1441.
[0328] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.
[0329] The audio circuit 1460 , the speaker 1461 , and the microphone 1462 can provide an audio interface between the user and the mobile phone.
[0330] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1470. It provides users with wireless broadband Internet access.
[0331] Processor 1480 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in memory 1420 and calling data stored in memory 1420.
[0332] The mobile phone also includes a power source 1490 (such as a battery) for supplying power to various components.
[0333] In this embodiment, the processor 1480 included in the terminal device also has the following functions:
[0334] Acquire an image to be desensitized, wherein the image to be desensitized includes content to be desensitized;
[0335] Determine an outer contour curve of the outer contour of the to-be-desensitized content in the to-be-desensitized image;
[0336] According to the outer contour curve, generating desensitized content whose shape conforms to the outer contour curve, wherein the desensitized content and the content to be desensitized have the same content type and different image styles;
[0337] A desensitized image is obtained by replacing the content to be desensitized in the image to be desensitized with the desensitized content.
[0338] Alternatively, in this embodiment, the processor 1480 included in the terminal device further has the following functions:
[0339] Acquiring a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle;
[0340] When it is determined that the vehicle environment image includes content to be desensitized, generating corresponding desensitized content according to the content to be desensitized, wherein the desensitized content has the same content type and a different image style as the content to be desensitized;
[0341] Obtaining a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content;
[0342] In response to an acquisition request sent by an external device for the vehicle environment image, the desensitized environment image is returned to the external device.
[0343] If the computer device is a server, the present application embodiment also provides a server, see Fig.17 As shown, Fig.17The structural diagram of the server 1500 provided in the embodiment of the present application, the server 1500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1522 (for example, one or more processors) and a memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the server 1500.
[0344] The server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server 2003 or Windows Server 2008. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.
[0345] The steps performed by the server in the above embodiment can be based on Fig.17 The server structure shown.
[0346] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided in the above embodiment.
[0347] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.
[0348] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.
[0349] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0350] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application. Moreover, based on the implementation methods provided in the above aspects, the present application can also be further combined to provide more implementation methods. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An image desensitization method, characterized in that: The method comprises: Acquire an image to be desensitized, wherein the image to be desensitized includes content to be desensitized; Determine an outer contour curve of the outer contour of the to-be-desensitized content in the to-be-desensitized image; According to the outer contour curve, generating desensitized content whose shape conforms to the outer contour curve, wherein the desensitized content and the content to be desensitized have the same content type and different image styles; Obtaining a desensitized image by replacing the to-be-desensitized content in the to-be-desensitized image with the desensitized content; The step of generating, according to the outer contour curve, desensitized content whose shape conforms to the outer contour curve comprises: According to the outer contour curve, generating desensitized content having a shape conforming to the outer contour curve through a content generation model; If the image style of the desensitized content is the target style, the content generation model is trained in the following manner: Acquire a training data set including contour-desensitized sample pairs, wherein the contour sample and the desensitized sample in the contour-desensitized sample pair have the same sample outer contour curve, and the desensitized sample and the desensitized content have the same image style; According to the contour sample, obtaining, by an initial generator, first predicted desensitized content having the same image style as the desensitized content; According to the contour sample, the desensitized sample and the first predicted desensitized content, a first discrimination result and a second discrimination result are obtained by an initial discriminator, wherein the first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style; Generate a first adversarial loss function based on the first discrimination result and the second discrimination result, and perform adversarial training on the initial generator and the initial discriminator through the first adversarial loss function; The initial generator is trained through the adversarial training to obtain the content generation model.
2. The method according to claim 1, characterized in that The sample outer contour curve is determined according to the outer contour of the content to be desensitized in the image sample, and the image styles of the image sample and the image to be desensitized are both source styles.
3. The method according to claim 1, characterized in that The performing adversarial training on the initial generator and the initial discriminator by using the adversarial loss function comprises: By fixing the model parameters of the initial discriminator, the model parameters of the initial generator are adjusted with minimizing the adversarial loss function as the optimization goal; By fixing the model parameters of the initial generator and taking maximizing the adversarial loss function as the optimization goal, the model parameters of the initial discriminator are adjusted.
4. The method according to claim 2, characterized in that: The training data set is obtained in the following way: Acquire the image sample, wherein the outer contour of the content to be desensitized in the image sample is marked with the sample outer contour curve; According to the image sample, obtaining the corresponding desensitized sample through a style conversion model; The training data set is constructed based on the sample outer contour curve corresponding to the image sample and the desensitized sample.
5. The method according to claim 4, characterized in that The style transfer model is trained in the following way: Obtain a desensitized image whose image style is the target style; According to the image sample, obtaining a second predicted desensitized content through an initial source-target generator; and according to the desensitized image, obtaining a predicted content to be desensitized through an initial target-source generator; According to the second predicted desensitized content and the desensitized image, a third discrimination result is obtained by an initial target style discriminator, wherein the third discrimination result is used to respectively identify whether the second predicted desensitized content and the desensitized image conform to the target style; and obtaining a fourth discrimination result through an initial source style discriminator according to the predicted content to be desensitized and the image sample, wherein the fourth discrimination result is used to respectively identify whether the predicted content to be desensitized and the image sample conform to the source style; generating an adversarial loss based on the third discrimination result and the fourth discrimination result; According to the second predicted desensitized content, reconstructing the to-be-desensitized content through the initial target-source generator; and obtaining reconstructed desensitized content through the initial source-target generator according to the predicted content to be desensitized; generating a first reconstruction difference loss for identifying a difference between the image sample and the reconstructed desensitized content, and generating a second reconstruction difference loss for identifying a difference between the desensitized content and the reconstructed desensitized content; generating a second adversarial loss function according to the adversarial loss, the first reconstruction difference loss, and the second reconstruction difference loss, and performing adversarial training on the initial source-target generator, the initial target-source generator, the initial target style discriminator, and the initial source style discriminator through the second adversarial loss function; The initial source-target generator is trained through the adversarial training to obtain the style transfer model.
6. The method according to claim 5, characterized in that The method of performing adversarial training on the initial source-target generator, the initial target-source generator, the initial target style discriminator, and the initial source style discriminator by using the second adversarial loss function comprises: By fixing the model parameters of the initial target style discriminator and the initial source style discriminator, the model parameters of the initial source-target generator and the initial target-source generator are adjusted with minimization of the second adversarial loss function as the optimization goal; By fixing the model parameters of the initial source-target generator and the initial target-source generator, the model parameters of the initial target style discriminator and the initial source style discriminator are adjusted with maximizing the second adversarial loss function as the optimization goal.
7. The method according to claim 5, characterized in that A context information extraction module is arranged between the encoder and decoder of the initial source-target generator and the initial target-source generator, and the context information extraction module includes dilated convolution layers with different dilation rates, and the dilated convolution layers with different dilation rates are used to improve the receptive field of the decoder.
8. The method according to claim 7, characterized in that The initial generator, the initial source-target generator and the initial target-source generator have the same model structure, and the initial discriminator, the initial target style discriminator and the initial source style discriminator have the same model structure.
9. The method according to claim 1, characterized in that: The step of determining an outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized includes: Performing desensitization content detection on the image to be desensitized, and obtaining a detection frame corresponding to the content to be desensitized in the image to be desensitized; Generate an expanded cropped image corresponding to the content to be desensitized according to the detection frame and the image to be desensitized; The expanded cropped image is segmented into the content to be desensitized to obtain an outer contour curve corresponding to the outer contour of the content to be desensitized.
10. The method according to claim 1, characterized in that The content types of the desensitized content and the content to be desensitized are face types, the image style of the desensitized content is anime style, and the image style of the content to be desensitized is reality style.
11. An image desensitization method, characterized in that: The method comprises: Acquiring a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle; When it is determined that the vehicle environment image includes content to be desensitized, generating desensitized content whose shape conforms to the outer contour curve according to the outer contour curve of the content to be desensitized, wherein the desensitized content has the same content type and a different image style as the content to be desensitized; Obtaining a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content; In response to an acquisition request sent by an external device for the vehicle environment image, returning the desensitized environment image to the external device; The step of generating, according to the outer contour curve, desensitized content whose shape conforms to the outer contour curve comprises: According to the outer contour curve, generating desensitized content having a shape conforming to the outer contour curve through a content generation model; If the image style of the desensitized content is the target style, the content generation model is trained in the following manner: Acquire a training data set including contour-desensitized sample pairs, wherein the contour sample and the desensitized sample in the contour-desensitized sample pair have the same sample outer contour curve, and the desensitized sample and the desensitized content have the same image style; According to the contour sample, obtaining, by an initial generator, first predicted desensitized content having the same image style as the desensitized content; According to the contour sample, the desensitized sample and the first predicted desensitized content, a first discrimination result and a second discrimination result are obtained by an initial discriminator, wherein the first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style; Generate a first adversarial loss function based on the first discrimination result and the second discrimination result, and perform adversarial training on the initial generator and the initial discriminator through the first adversarial loss function; The initial generator is trained through the adversarial training to obtain the content generation model.
12. The method according to claim 11, characterized in that After obtaining a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content, the method further includes: The vehicle environment image is deleted at the vehicle-mounted terminal.
13. The method according to claim 11, characterized in that The vehicle environment image is a video frame in the vehicle exterior environment video captured by the vehicle-mounted terminal.
14. The method according to claim 11, characterized in that The method further comprises: When it is determined that an abnormal event occurs for the target vehicle, a prompt message is sent to the external device, wherein the vehicle environment image is collected by the vehicle-mounted terminal when the abnormal event occurs; The acquisition request sent by the external device based on the prompt information is acquired.
15. An image desensitization device, characterized in that: The device comprises an acquisition unit, a determination unit, a generation unit and a replacement unit: The acquisition unit is used to acquire the image to be desensitized, wherein the image to be desensitized includes the content to be desensitized; The determining unit is used to determine an outer contour curve of the outer contour of the content to be desensitized in the image to be desensitized; The generating unit is configured to generate, according to the outer contour curve, desensitized content having a shape conforming to the outer contour curve, wherein the desensitized content and the content to be desensitized have the same content type and a different image style; The replacing unit is used to obtain a desensitized image by replacing the to-be-desensitized content in the to-be-desensitized image with the desensitized content; The generating unit is further configured to: According to the outer contour curve, generating desensitized content having a shape conforming to the outer contour curve through a content generation model; If the image style of the desensitized content is the target style, the device further includes a first training unit, and the first training unit is used to train the content generation model in the following manner: Acquire a training data set including contour-desensitized sample pairs, wherein the contour sample and the desensitized sample in the contour-desensitized sample pair have the same sample outer contour curve, and the desensitized sample and the desensitized content have the same image style; According to the contour sample, obtaining, by an initial generator, first predicted desensitized content having the same image style as the desensitized content; According to the contour sample, the desensitized sample and the first predicted desensitized content, a first discrimination result and a second discrimination result are obtained by an initial discriminator, wherein the first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style; Generate a first adversarial loss function based on the first discrimination result and the second discrimination result, and perform adversarial training on the initial generator and the initial discriminator through the first adversarial loss function; The initial generator is trained through the adversarial training to obtain the content generation model.
16. The device according to claim 15, characterized in that The sample outer contour curve is determined according to the outer contour of the content to be desensitized in the image sample, and the image styles of the image sample and the image to be desensitized are both source styles.
17. The device according to claim 15, characterized in that The first training unit is also used for: By fixing the model parameters of the initial discriminator, the model parameters of the initial generator are adjusted with minimizing the adversarial loss function as the optimization goal; By fixing the model parameters of the initial generator and taking maximizing the adversarial loss function as the optimization goal, the model parameters of the initial discriminator are adjusted.
18. The device according to claim 16, characterized in that The acquisition unit is also used for: Acquire the image sample, wherein the outer contour of the content to be desensitized in the image sample is marked with the sample outer contour curve; According to the image sample, obtaining the corresponding desensitized sample through a style conversion model; The training data set is constructed based on the sample outer contour curve corresponding to the image sample and the desensitized sample.
19. The device according to claim 18, characterized in that The device further includes a second training unit, and the second training unit is trained to obtain the style conversion model in the following manner: Obtain a desensitized image whose image style is the target style; According to the image sample, obtaining a second predicted desensitized content through an initial source-target generator; and according to the desensitized image, obtaining a predicted content to be desensitized through an initial target-source generator; According to the second predicted desensitized content and the desensitized image, a third discrimination result is obtained by an initial target style discriminator, wherein the third discrimination result is used to respectively identify whether the second predicted desensitized content and the desensitized image conform to the target style; and obtaining a fourth discrimination result through an initial source style discriminator according to the predicted content to be desensitized and the image sample, wherein the fourth discrimination result is used to respectively identify whether the predicted content to be desensitized and the image sample conform to the source style; generating an adversarial loss based on the third discrimination result and the fourth discrimination result; According to the second predicted desensitized content, reconstructing the to-be-desensitized content through the initial target-source generator; and obtaining reconstructed desensitized content through the initial source-target generator according to the predicted content to be desensitized; generating a first reconstruction difference loss for identifying a difference between the image sample and the reconstructed desensitized content, and generating a second reconstruction difference loss for identifying a difference between the desensitized content and the reconstructed desensitized content; generating a second adversarial loss function according to the adversarial loss, the first reconstruction difference loss, and the second reconstruction difference loss, and performing adversarial training on the initial source-target generator, the initial target-source generator, the initial target style discriminator, and the initial source style discriminator through the second adversarial loss function; The initial source-target generator is trained through the adversarial training to obtain the style transfer model.
20. The device according to claim 19, characterized in that The second training unit is also used for: By fixing the model parameters of the initial target style discriminator and the initial source style discriminator, the model parameters of the initial source-target generator and the initial target-source generator are adjusted with minimization of the second adversarial loss function as the optimization goal; By fixing the model parameters of the initial source-target generator and the initial target-source generator, the model parameters of the initial target style discriminator and the initial source style discriminator are adjusted with maximizing the second adversarial loss function as the optimization goal.
21. The device according to claim 19, characterized in that A context information extraction module is arranged between the encoder and decoder of the initial source-target generator and the initial target-source generator, and the context information extraction module includes dilated convolution layers with different dilation rates, and the dilated convolution layers with different dilation rates are used to improve the receptive field of the decoder.
22. The device according to claim 21, characterized in that The initial generator, the initial source-target generator and the initial target-source generator have the same model structure, and the initial discriminator, the initial target style discriminator and the initial source style discriminator have the same model structure.
23. The device according to claim 15, characterized in that The determining unit is further configured to: Performing desensitization content detection on the image to be desensitized, and obtaining a detection frame corresponding to the content to be desensitized in the image to be desensitized; Generate an expanded cropped image corresponding to the content to be desensitized according to the detection frame and the image to be desensitized; The expanded cropped image is segmented into the content to be desensitized to obtain an outer contour curve corresponding to the outer contour of the content to be desensitized.
24. The device according to claim 15, characterized in that The content types of the desensitized content and the content to be desensitized are face types, the image style of the desensitized content is anime style, and the image style of the content to be desensitized is reality style.
25. An image desensitization device, characterized in that: The device comprises an acquisition unit, a generation unit, a replacement unit and a sending unit: The acquisition unit is used to acquire a vehicle environment image corresponding to the target vehicle through a vehicle-mounted terminal configured in the target vehicle; The generating unit is configured to generate desensitized content having a shape conforming to the outer contour curve of the content to be desensitized when it is determined that the vehicle environment image includes the content to be desensitized, wherein the desensitized content and the content to be desensitized have the same content type and a different image style; The replacing unit is used to obtain a desensitized environment image corresponding to the vehicle environment image by replacing the to-be-desensitized content in the vehicle environment image with the desensitized content; The sending unit is used to return the desensitized environment image to the external device in response to an acquisition request sent by the external device for the vehicle environment image; The step of generating, according to the outer contour curve, desensitized content whose shape conforms to the outer contour curve comprises: According to the outer contour curve, generating desensitized content having a shape conforming to the outer contour curve through a content generation model; If the image style of the desensitized content is the target style, the content generation model is trained in the following manner: Acquire a training data set including contour-desensitized sample pairs, wherein the contour sample and the desensitized sample in the contour-desensitized sample pair have the same sample outer contour curve, and the desensitized sample and the desensitized content have the same image style; According to the contour sample, obtaining, by an initial generator, first predicted desensitized content having the same image style as the desensitized content; According to the contour sample, the desensitized sample and the first predicted desensitized content, a first discrimination result and a second discrimination result are obtained by an initial discriminator, wherein the first discrimination result is used to identify whether the shape of the desensitized sample conforms to the sample outer contour curve and whether the desensitized sample conforms to the target style, and the second discrimination result is used to identify whether the shape of the first predicted desensitized content conforms to the sample outer contour curve and whether the first predicted desensitized content conforms to the target style; Generate a first adversarial loss function based on the first discrimination result and the second discrimination result, and perform adversarial training on the initial generator and the initial discriminator through the first adversarial loss function; The initial generator is trained through the adversarial training to obtain the content generation model.
26. The device according to claim 25, characterized in that The device further includes a deleting unit, which is used to delete the vehicle environment image in the vehicle-mounted terminal.
27. The device according to claim 25, characterized in that The vehicle environment image is a video frame in the vehicle exterior environment video captured by the vehicle-mounted terminal.
28. The device according to claim 25, characterized in that The sending unit is further used to send prompt information to the external device when it is determined that an abnormal event occurs for the target vehicle, and the vehicle environment image is collected by the vehicle terminal when the abnormal event occurs; The acquisition unit is further configured to acquire the acquisition request sent by the external device based on the prompt information.
29. A computer device, characterized in that: The computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the method according to any one of claims 1 to 14 according to the instructions in the program code.
30. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the method according to any one of claims 1 to 14.
31. A computer program product comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 14.
