A method and device for determining confidence of segmentation image

By segmenting the images in the detection frame, the confidence level is adjusted to reflect the quality of the segmented map, the error display problem caused by low confidence in the prior art is solved, and the confidence level of the segmented map and the display quality of AR special effects is improved.

CN113066048BActive Publication Date: 2024-11-05HUAWEI TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110221912.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-02-27
Publication Date
2024-11-05
Estimated Expiration
2041-02-27

AI Technical Summary

Technical Problem

When the existing segmentation algorithms represent the confidence of the detection box, they cannot effectively express the image segmentation quality of the segmentation map, resulting in objects with low confidence in applications such as AR special effects display and other applications are also mistakenly displayed for special effects.

Method used

By segmenting the images in the detection frame, the image segmentation quality of the segmented image is obtained, and the confidence of the detection frame is adjusted based on the quality to obtain more accurate confidence in the segmented image, ensuring that only high-quality segmented images show special effects .

Benefits of technology

The accuracy of the confidence of segmented images is improved, so that the special effects are only displayed on high-quality target objects, reducing virtual and real occlusion phenomena, and improving the display effect of AR special effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113066048B_ABST
    Figure CN113066048B_ABST
Patent Text Reader

Abstract

The present application discloses a method for determining the confidence of a segmentation map, the method comprising: obtaining the confidence of a detection frame of a target object in an image, then obtaining the image segmentation quality of the segmentation map of the target object, and adjusting the aforementioned confidence according to the image segmentation quality to obtain the adjusted confidence as the confidence of the segmentation map. It can be seen that the confidence of the detection frame is adjusted using the image segmentation quality of the segmentation map to obtain the confidence of the segmentation map, so that the confidence of the segmentation map can include the segmentation quality information of the segmentation map itself, thereby improving the accuracy of the confidence of the segmentation map.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for determining the confidence of a segmentation image. Background Art

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras / camcorders and computers to obtain the data and information of the objects being photographed that people need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track, and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a discipline about how to use cameras / camcorders and computers to obtain the data and information of the objects being photographed that people need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track, and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously. Among them, image segmentation is a commonly used technology in the field of computer vision.

[0004] At present, instance-related segmentation algorithms based on deep learning technology have been widely used in the industry. Generally speaking, the target detection network is used to output the category, detection box and segmentation map of the target object. There will be a corresponding confidence for the detection box. In actual applications, the detection boxes will be selected for the next application based on the confidence. In some applications, it is necessary to obtain the confidence of the segmentation map. For example, in the application of AR special effects display, it is necessary to display AR special effects on objects with high segmentation map confidence, and not on objects with low segmentation map confidence.

[0005] In existing implementations, the confidence of the detection box is directly reused as the confidence of the segmentation map. Since the confidence of the detection box can only represent the positioning confidence of the detection box, it cannot well express the image segmentation quality of the segmentation map. Summary of the invention

[0006] The present application provides a segmentation map confidence determination method, device, etc. to improve the accuracy of the segmentation map confidence. The present application is introduced below through different aspects. It can be understood that the implementation methods and beneficial effects of the following aspects can refer to each other.

[0007] In a first aspect, the present application provides a method for determining confidence of a segmentation map, the method comprising:

[0008] Get the first image. In one implementation, the terminal device can obtain the input video stream, wherein the video stream can be captured by the shooting device in the terminal device. Specifically, in the scene where AR special effects are displayed, the video stream can be a real-time video stream captured by the shooting device of the terminal device. Among them, the video stream includes a first image (or a first image frame). In one implementation, the user can select a video stream on the terminal device, wherein the selected video stream includes the first image. In one implementation, the terminal device can obtain the input first image, wherein the first image can be captured by the shooting device in the terminal device. In one implementation, the user can select the first image on the terminal device (for example, in an album on the terminal device or an album on the cloud side).

[0009] Performing target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence of the detection frame, where the first confidence is used to indicate a positioning accuracy of the detection frame;

[0010] Among them, a pre-trained neural network model can be used to perform target detection on the first image to obtain a target detection result of the target object, wherein the target detection result may include a detection box of the target object, and accordingly, the first confidence level is used to indicate the positioning accuracy of the detection box.

[0011] Performing image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and to obtain image segmentation quality of the segmentation map;

[0012] Among them, a pre-trained neural network model can also be used to perform image segmentation on the image in the detection frame to obtain a segmentation map corresponding to the target object.

[0013] The first confidence level is adjusted according to the image segmentation quality to obtain a second confidence level of the segmentation map, wherein the higher the image segmentation quality is, the greater the second confidence level is.

[0014] In order to calculate the confidence of the segmentation map, the image segmentation quality of the segmentation map can be obtained, and the first confidence of the detection frame can be adjusted based on the image segmentation quality, so that the adjusted first confidence can indicate not only the positioning accuracy of the detection frame but also the image segmentation quality of the segmentation map. In one implementation, the image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation map, wherein the higher the boundary clarity, the higher the image segmentation quality.

[0015] This embodiment uses the image segmentation quality of the segmentation map to adjust the first confidence of the detection frame to obtain a second confidence as the confidence of the segmentation map, so that the second confidence includes not only the positioning accuracy of the detection frame, but also the segmentation quality information of the segmentation map itself, so as to obtain a more accurate segmentation map confidence. In addition, the segmentation map confidence determination method in this embodiment does not require the addition of an additional network to calculate the confidence of the segmentation map. Based on the results of target detection and the results of the segmentation map, the accurate segmentation map confidence can be directly obtained without adding a lot of additional calculations. It is more friendly to the end side, making the solution easy to deploy on the end side for implementation.

[0016] In a possible implementation, the image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation image, wherein the higher the boundary clarity, the higher the image segmentation quality.

[0017] Taking the target object as a person object as an example, the value of each pixel point of the segmentation map can be between 0 and 1, among which the pixel value of the pixel point in the background area is 0 or close to 0, and the pixel value of the pixel point in the boundary area of ​​the target object is around 0.5. When using the segmentation map, a 0.5 threshold can be used for binarization, so errors will occur at the edge. Ideally, the fewer pixel points with pixel values ​​near 0.5 at the edge of the human body in a segmentation map, the better the quality of the segmentation map.

[0018] In a possible implementation, the second confidence of the segmentation map is used to indicate the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, wherein the higher the positioning accuracy is, the higher the second confidence is.

[0019] In a possible implementation, the segmentation map includes an object region where the target object is located, a background region of the target object, and a boundary region between the object region and the background region, and acquiring the image segmentation quality of the segmentation map includes:

[0020] The ratio of the number of pixels included in the object area to the number of pixels included in the target area is used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area.

[0021] In the embodiment of the present application, the ratio of the number of pixels included in the object area to the number of pixels included in the target area can be used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area. In other words, the smaller the boundary area (or the fewer the number of pixels in the boundary area), the higher the image segmentation quality.

[0022] In a possible implementation, the method further includes:

[0023] Determine the area where the pixels in the segmentation map whose pixel values ​​are greater than the foreground threshold are located as the object area;

[0024] Determine the area where the pixels in the segmentation map whose pixel values ​​are less than the background threshold are located as the background area;

[0025] The region where the pixel points in the segmentation image whose pixel values ​​are greater than the background threshold and less than the foreground threshold are located is determined as the boundary region.

[0026] Specifically, the segmentation map includes an object area where the target object is located, a background area of ​​the target object, and a boundary area between the object area and the background area. The area where the pixel points in the segmentation map have pixel values ​​greater than a foreground threshold value are located can be determined as the object area, the area where the pixel points in the segmentation map have pixel values ​​less than a background threshold value are located can be determined as the background area, and the area where the pixel points in the segmentation map have pixel values ​​greater than the background threshold value and less than the foreground threshold value are located can be determined as the boundary area. For example, the background threshold value can be 0.25, then the pixel values ​​of the pixels in the background area are less than 0.25, and the foreground threshold value can be 0.7, then the pixel values ​​of the pixels in the foreground area are greater than 0.7, and then the pixel values ​​of the pixels in the boundary area are greater than 0.25 and less than 0.7.

[0027] In a possible implementation, obtaining the image segmentation quality of the segmentation map includes:

[0028] Acquire initial pixel values ​​of some pixels included in the segmentation map, wherein the initial pixel values ​​of some pixels are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1;

[0029] According to the target mapping relationship, the initial pixel value is adjusted to obtain an adjusted pixel value of the pixel point included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, and in the target mapping relationship, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases;

[0030] The image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

[0031] In a possible implementation, obtaining the image segmentation quality of the segmentation map includes:

[0032] Acquire initial pixel values ​​of some pixels included in the segmentation map, wherein the initial pixel values ​​of some pixels are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1;

[0033] According to the target mapping relationship, the initial pixel value is adjusted to obtain an adjusted pixel value of the pixel point included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, and in the target mapping relationship, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases;

[0034] The image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

[0035] In this embodiment, the pixel values ​​of the pixels in the segmentation map are mapped based on a preset mapping relationship, wherein the image segmentation quality of the segmentation map is expressed based on the average value of the mapped pixel values, which is equivalent to using the mapped pixel values ​​as the weights when calculating the image segmentation quality of the pixels. The larger the weight, the greater the effect of improving the image segmentation quality, and the smaller the weight, the greater the effect of reducing the image segmentation quality. Specifically, for the pixel points whose pixel values ​​are less than the foreground threshold and greater than or equal to the background threshold, they are the cause of the low image segmentation quality of the segmentation map, so the weight assigned to this part of the pixels is very small, for example, a value of 0 or close to 0 is assigned. Therefore, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases, that is, at the initial pixel value close to the preset value, the assigned weight is close to 0, and as the initial pixel value increases, the assigned weight gradually increases. When the initial pixel value is 1, the assigned weight reaches 1.

[0036] In this embodiment, as the initial pixel value increases, the slope of the target mapping relationship may gradually increase, so the target mapping relationship is equivalent to a mapping relationship that becomes increasingly steeper.

[0037] In a possible implementation, adjusting the first confidence level according to the image segmentation quality to obtain a second confidence level of the segmentation map includes:

[0038] A product of the image segmentation quality and the first confidence is determined as a second confidence of the segmentation map.

[0039] Through the above method, the second confidence of the segmentation map can indicate both the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, wherein the higher the positioning accuracy, the higher the second confidence, and the higher the positioning accuracy of the detection box, the higher the second confidence.

[0040] In a possible implementation, after acquiring the second confidence level of the segmentation map, the method further includes:

[0041] Based on the second confidence being higher than a threshold, displaying an AR object around the target object in the first image; or,

[0042] Based on the second confidence being higher than a threshold, the target object in the first image is replaced with a first object, the first object being different from the target object.

[0043] In the scenario of AR special effects display, the terminal device can display an AR object around the target object in the first image based on the second confidence level being higher than a threshold. Specifically, the AR object can be displayed around the target object based on the 3D position information of the target object.

[0044] In the AI ​​pedestrian removal scenario, if the target object is a determined moving pedestrian, the terminal device can replace the target object with a first object based on the second confidence level being higher than a threshold, wherein the first object is the background area where the target object is located in other image frames (that is, in the first image, area A is the area where the target object is located, and in other image frames, area A is the background area where the target object is located, and the target object has moved to another position in the image or is not in the image).

[0045] In a second aspect, the present application provides a segmentation map confidence determination device, the device comprising:

[0046] An acquisition module, used for acquiring a first image;

[0047] a target detection module, configured to perform target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence level of the detection frame, wherein the first confidence level is used to indicate a positioning accuracy of the detection frame;

[0048] An image segmentation module is used to perform image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and to obtain the image segmentation quality of the segmentation map;

[0049] A confidence determination module is used to adjust the first confidence according to the image segmentation quality to obtain a second confidence of the segmentation map, wherein the higher the image segmentation quality, the greater the second confidence.

[0050] Among them, the first confidence of the detection frame is adjusted using the image segmentation quality of the segmentation map to obtain the second confidence of the segmentation map, so that the second confidence includes not only the positioning accuracy of the detection frame, but also the segmentation quality information of the segmentation map itself, so that a more accurate segmentation map confidence can be obtained. In addition, the segmentation map confidence determination method in this embodiment does not require the addition of an additional network to calculate the confidence of the segmentation map. Based on the results of target detection and the results of the segmentation map, an accurate segmentation map confidence can be directly obtained without adding a lot of additional calculations. It is more friendly to the end side, making the solution easy to deploy on the end side for implementation.

[0051] In a possible implementation, the image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation image, wherein the higher the boundary clarity, the higher the image segmentation quality.

[0052] In a possible implementation, the second confidence of the segmentation map is used to indicate the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, wherein the higher the positioning accuracy is, the higher the second confidence is.

[0053] In a possible implementation, the segmentation map includes an object region where the target object is located, a background region of the target object, and a boundary region between the object region and the background region, and the acquisition module is used to:

[0054] The ratio of the number of pixels included in the object area to the number of pixels included in the target area is used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area.

[0055] In a possible implementation, the acquisition module is used to: determine the area where the pixels in the segmentation map whose pixel values ​​are greater than the foreground threshold are located as the object area; determine the area where the pixels in the segmentation map whose pixel values ​​are less than the background threshold are located as the background area; determine the area where the pixels in the segmentation map whose pixel values ​​are greater than the background threshold and less than the foreground threshold are located as the boundary area.

[0056] In a possible implementation, the image segmentation module is used to perform image segmentation on the image within the detection frame to obtain an initial segmentation map corresponding to the target object, wherein the initial segmentation map includes multiple pixels and the probability that each pixel belongs to each category; the maximum probability value among the probabilities that each pixel belongs to each category is used as the pixel value of each pixel in the segmentation map; the pixel values ​​of some pixels included in the segmentation map are obtained, wherein the pixel values ​​of some pixels are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1; the image segmentation quality of the segmentation map is determined according to the pixel values ​​of some pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map, the higher the image segmentation quality.

[0057] In a possible implementation, the image segmentation module is used to obtain initial pixel values ​​of some pixels included in the segmentation map, wherein the initial pixel values ​​of some pixels are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1; according to a target mapping relationship, the initial pixel values ​​are adjusted to obtain adjusted pixel values ​​of the pixels included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, in which, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases; the image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map, the higher the image segmentation quality.

[0058] In a possible implementation, the confidence determination module is used to determine a product of the image segmentation quality and the first confidence as a second confidence of the segmentation map.

[0059] In one possible implementation, the device also includes: an image processing module, used to display an AR object around the target object in the first image based on the second confidence being higher than a threshold; or, based on the second confidence being higher than a threshold, replacing the target object in the first image with a first object, which is different from the target object.

[0060] In a third aspect, an embodiment of the present application provides a model training device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to perform any optional method such as the first aspect above.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect, third aspect and any optional method thereof.

[0062] In a fifth aspect, an embodiment of the present application provides a computer program, including codes, which, when executed, execute any optional method of the first aspect described above.

[0063] In a sixth aspect, the present application provides a chip system, which includes a processor for supporting an execution device or a training device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or, information. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or the training device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0064] The embodiment of the present application provides a method for determining the confidence of a segmentation map, the method comprising: acquiring a first image; performing target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence of the detection frame, the first confidence being used to indicate the positioning accuracy of the detection frame; performing image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and to obtain the image segmentation quality of the segmentation map; adjusting the first confidence according to the image segmentation quality to obtain a second confidence of the segmentation map, wherein the higher the image segmentation quality, the greater the second confidence. In the above manner, the first confidence of the detection frame is adjusted using the image segmentation quality of the segmentation map to obtain the second confidence of the segmentation map, thereby making the second confidence include not only the positioning accuracy of the detection frame, but also the segmentation quality information of the segmentation map itself, so as to obtain a segmentation map confidence with higher accuracy. Moreover, the segmentation map confidence determination method in this embodiment does not require the addition of an additional network to calculate the confidence of the segmentation map. The accurate segmentation map confidence can be directly obtained based on the results of target detection and the results of the segmentation map. There is no need to add a lot of additional calculations, which is more friendly to the terminal side, making the solution easy to deploy on the terminal side for implementation. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0066] Figure 2a is a software structure block diagram of a terminal according to an embodiment of the present disclosure;

[0067] Figure 2b A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0068] Figure 2c A segmentation map confidence determination system is also provided for the embodiment of the present application;

[0069] Figure 3 An illustration of an application scenario provided by an embodiment of the present application;

[0070] Figure 4 An illustration of an application scenario provided by an embodiment of the present application;

[0071] Figure 5 A schematic diagram of a flow chart of a method for determining the confidence of a segmentation image provided in an embodiment of the present application;

[0072] Figure 6 The structure of a backbone network is shown in FIG.

[0073] Figure 7 It is a hint of a header;

[0074] Figure 8 A schematic diagram of a segmentation diagram of an embodiment of the present application;

[0075] Fig. 9 It is a schematic diagram of various target mapping relationships in this embodiment;

[0076] Fig.10 A schematic diagram of the structure of a segmentation image confidence determination device provided in an embodiment of the present application;

[0077] Fig.11 A schematic diagram of the structure of a server provided in an embodiment of the present application;

[0078] Fig.12 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0080] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0081] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0082] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0083] For ease of understanding, the structure of the terminal 100 provided in the embodiment of the present application is described below by way of example. Figure 1 , Figure 1 It is a schematic diagram of the structure of the terminal device provided in an embodiment of the present application.

[0084] like Figure 1 As shown, the terminal 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0085] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0086] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0087] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0088] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0090] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal 100.

[0091] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0092] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0093] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0094] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal 100.

[0095] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0096] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal 100, and can also be used to transmit data between the terminal 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0097] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal 100. In other embodiments of the present application, the terminal 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0098] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0099] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0100] The wireless communication function of the terminal 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0101] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0102] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied on the terminal 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0103] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0104] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0105] In some embodiments, the antenna 1 of the terminal 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0106] The terminal 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0107] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0108] The terminal 100 can realize the shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194 and the application processor.

[0109] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0110] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0111] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0112] Video codecs are used to compress or decompress digital videos. Terminal 100 may support one or more video codecs. Thus, terminal 100 may play or record videos in various coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0113] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0114] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0115] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0116] The terminal 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor.

[0117] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0118] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0119] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0120] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by putting his mouth close to microphone 170C to input the sound signal into microphone 170C. Terminal 100 may be provided with at least one microphone 170C. In other embodiments, terminal 100 may be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, terminal 100 may also be provided with three, four or more microphones 170C to realize collection of sound signals, noise reduction, identification of sound source, realization of directional recording function, etc.

[0121] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0122] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal 100 detects the touch operation intensity according to the pressure sensor 180A. The terminal 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0123] The gyroscope sensor 180B can be used to determine the motion posture of the terminal 100. In some embodiments, the angular velocity of the terminal 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. Exemplarily, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0124] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0125] The magnetic sensor 180D includes a Hall sensor. The terminal 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal 100 is a flip phone, the terminal 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0126] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal 100 in various directions (generally three axes). When the terminal 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0127] The distance sensor 180F is used to measure the distance. The terminal 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0128] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal 100 emits infrared light outward through the light emitting diode. The terminal 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal 100. When insufficient reflected light is detected, the terminal 100 can determine that there is no object near the terminal 100. The terminal 100 can use the proximity light sensor 180G to detect that the user holds the terminal 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0129] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal 100 is in a pocket to prevent accidental touches.

[0130] The fingerprint sensor 180H is used to collect fingerprints. The terminal 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0131] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal 100 heats the battery 142 to avoid abnormal shutdown of the terminal 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0132] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal 100, which is different from the position of the display screen 194.

[0133] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0134] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal 100 may receive key input and generate key signal input related to user settings and function control of the terminal 100.

[0135] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0136] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0137] The SIM card interface 195 is used to connect the SIM card. The SIM card can be connected to and separated from the terminal 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal 100 interacts with the network through the SIM card to realize functions such as calls and data communications. In some embodiments, the terminal 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal 100 and cannot be separated from the terminal 100.

[0138] The software system of the terminal 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the terminal 100.

[0139] Figure 2a It is a software structure block diagram of the terminal 100 according to an embodiment of the present disclosure.

[0140] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0141] The application layer can include a series of application packages.

[0142] like Figure 2a As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0143] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0144] like Figure 2a As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0145] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0146] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0147] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0148] The phone manager is used to provide communication functions of the terminal 100, such as management of call status (including connection, disconnection, etc.).

[0149] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0150] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0151] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0152] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0153] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0154] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0155] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0156] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0157] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0158] A 2D graphics engine is a drawing engine for 2D drawings.

[0159] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0160] The following is an illustrative description of the software and hardware workflow of the terminal 100 in conjunction with the capture and photo shooting scene.

[0161] When the touch sensor 180K receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a touch single-click operation and the control corresponding to the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0162] The embodiment of the present application also provides a server 1300.

[0163] The server 1300 may include a processor 1310 and a transceiver 1320. The transceiver 1320 may be connected to the processor 1310. Figure 2bAs shown. The transceiver 1320 may include a receiver and a transmitter, which may be used to receive or send messages or data, and the transceiver 1320 may be a network card. The server 1300 may also include an acceleration component (which may be called an accelerator). When the acceleration component is a network acceleration component, the acceleration component may be a network card. The processor 1310 may be the control center of the server 1300, and various interfaces and lines are used to connect various parts of the entire server 1300, such as the transceiver 1320. In the present invention, the processor 1310 may be a central processing unit (CPU), and optionally, the processor 1310 may include one or more processing units. The processor 1310 may also be a digital signal processor, an application-specific integrated circuit, a field programmable gate array, a GPU or other programmable logic devices. The server 1300 may also include a memory 1330, which may be used to store software programs and modules. The processor 1310 reads the software code and modules stored in the memory 1330 to execute various functional applications and data processing of the server 1300.

[0164] The present application also provides a segmentation map confidence determination system. Figure 2c As shown, the system may include a terminal device and a server. The terminal device may be a mobile terminal, a human-computer interaction device, or an in-vehicle visual perception device, such as a mobile phone, an intelligent robot, an unmanned vehicle, an intelligent monitor, an augmented reality (AR) wearable device, etc. Accordingly, the method provided in the embodiment of the present disclosure may be used in application fields such as human-computer interaction, in-vehicle visual perception, augmented reality, intelligent monitoring, and unmanned driving.

[0165] Exemplarily, the terminal device can send an image (such as the first image in the embodiment of the present application) to the server, which performs image processing and analysis to obtain the confidence of the segmentation map (such as the second confidence in the embodiment of the present application) and transmit the second confidence to the terminal device.

[0166] For ease of understanding, a method for determining the confidence of a segmentation map provided in an embodiment of the present application is specifically described in combination with the accompanying drawings and application scenarios.

[0167] First, several application scenario examples of the embodiments of the present application are described:

[0168] 1. AR special effects display

[0169] In this scenario, special effects can be added to the character objects in the video, such as displaying wings on the back of the character (such as Figure 3As shown); in a specific implementation, the video shot by the terminal device can be obtained, and the image frames in the video can be segmented to obtain a segmentation map for the character object, and based on the segmentation map and the position information of the character object in the image, special effects are displayed at the corresponding position.

[0170] In some cases, due to image quality or interference from obstructions, the recognition of human objects in the segmentation map may not be accurate enough. Therefore, it is necessary to obtain the confidence of the segmentation map and determine whether to display special effects based on the confidence of the segmentation map. If the confidence is low, it means that the object in the segmentation map may not be a human being, for example, it may be an object other than a human being, or the object in the segmentation map is a human object, but the position, outline (or boundary) and other information of the human object cannot be accurately determined from the segmentation map. In this case, special effects will not be displayed.

[0171] 2. AI goes to passers-by

[0172] In this scene, the main character and the passers-by in the background are segmented, and the high-confidence segmentation map is selected. The image area where the moving passers-by are located is cut out and replaced with the background area at the corresponding position in other image frames. Figure 4 As shown, Figure 4 The images 1401 and 1402 in the image 1401 are images that can be used for fusion processing. The moving target 1411 can be segmented from the image 1401 first, and then the image 1401 is completed by the background image 1412 of the image 1402 at the position of the removed moving target. It can be understood that when the clearest image is completed by the background image of other images at the position of the removed moving target, the background image of other images at the position of the removed moving target needs not to contain the moving target, for example, Figure 4 The background image 1412 at the position of the removed moving target in the image 1402 does not contain the moving target. It can be understood that the images used for fusion processing can be two images or more images. When there are many moving targets to be removed in the image, and two images are not enough to be fused to obtain a complete image without moving targets, more images can be fused to obtain a complete image without moving targets.

[0173] 3. Vehicle Object Segmentation in Autonomous Driving

[0174] In autonomous driving, it is necessary to provide instance segmentation results of road vehicles to the on-board terminal equipment to assist the autonomous driving system in making better driving decisions.

[0175] For example, in advanced driver assistance systems (ADAS) and advanced driver systems (ADS), multiple types of target detection are required in real time, including: dynamic obstacles (pedestrians, cyclists, tricycles, cars, trucks, buses), static obstacles (traffic cones, traffic sticks, fire hydrants, motorcycles, bicycles), traffic signs (TrafficSign, GuideSign, billboards, red traffic lights (TrafficLight_Red) / yellow traffic lights (TrafficLight_Yellow) / green traffic lights (TrafficLight_Green) / black traffic lights (TrafficLight_Black), road signs (RoadSign)). In order to fuse with the lidar data, it is necessary to obtain the mask of the dynamic obstacle, so as to filter out the laser point cloud that hits the dynamic obstacle.

[0176] The existing method for determining the confidence of a segmentation map can obtain the detection result of the object (including the detection frame and the classification result) after performing target detection on an image, and directly use the confidence of the detection result as the confidence of the segmentation map. Since the confidence of the detection result is used to indicate the positioning accuracy of the detection frame, the confidence can only express whether the position of the detection frame is located at the position of the object, but cannot reflect the segmentation map quality of the segmentation map.

[0177] Taking AR special effects display as an example, in the scene of AR special effects display, it may be necessary to obtain the clear position of the boundary of the target object in the segmentation map. Therefore, when the position of the detection frame is very accurate and the quality of the segmentation map is very low, the existing technology will also display AR special effects on the object. However, in this case, due to the low quality of the segmentation map, the effect of the displayed AR special effects (such as display position, display direction, etc.) is very poor. Taking the AR special effects of wings as an example, in this case, it will affect the mutual occlusion relationship between the wings and the human object, that is, the so-called virtual and real occlusion will occur. Therefore, in addition to including information on the positioning accuracy of the detection frame, the confidence of the segmentation map should also include the segmentation quality of the object in the segmentation map, such as whether the boundary is clear.

[0178] Reference Figure 5 , Figure 5An example of a method for determining the confidence level of a segmentation image provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the method for determining the confidence of a segmentation map provided in an embodiment of the present application includes:

[0179] 501. Acquire a first image;

[0180] The execution subject of step 501 may be a terminal device or a server. Specifically, the terminal device may obtain the first image; or the terminal device may obtain the first image and send the first image to a server on the cloud side, and then the server may obtain the first image.

[0181] Take the execution subject as a terminal device as an example.

[0182] In one implementation, the terminal device can obtain an input video stream, wherein the video stream can be captured by a camera in the terminal device. In a scene where AR special effects are displayed, the video stream is a real-time video stream captured by the camera of the terminal device, wherein the video stream includes a first image (or a first image frame).

[0183] In one implementation, a user may select a video stream on a terminal device, wherein the selected video stream includes a first image.

[0184] In one implementation, the terminal device may acquire an input first image, wherein the first image may be captured by a capturing device in the terminal device.

[0185] In one implementation, the user may select the first image on the terminal device (eg, in a photo album on the terminal device or a photo album on the cloud side).

[0186] In the embodiment of the present application, from the user's visual point of view, the first image may include a target object, wherein the target object may be a person object or a non-person object. The first image may include multiple objects, and the target object is one of the multiple objects.

[0187] It should be understood that after the first image is acquired, the first image may be pre-processed to stimulate normalization processing in the RGB domain, which is not limited in the present application.

[0188] 502. Perform target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence of the detection frame, where the first confidence is used to indicate a positioning accuracy of the detection frame.

[0189] 503. Perform image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and obtain image segmentation quality of the segmentation map.

[0190] In an embodiment of the present application, a pre-trained neural network model can be used to perform target detection on the first image to obtain a target detection result of the target object, wherein the target detection result may include a detection box of the target object, and accordingly, the first confidence level is used to indicate the positioning accuracy of the detection box.

[0191] In an embodiment of the present application, a pre-trained neural network model may also be used to perform image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object.

[0192] It should be understood that the segmentation map in the embodiment of the present application can also be referred to as a Mask or a mask activation map.

[0193] Exemplarily, the neural network model for target detection may include a backbone network and a headend Header, see Figure 6 , Figure 6 This is a schematic diagram of the structure of a backbone network, such as Figure 6 As shown in the figure, the backbone network is used to receive a first input image, perform convolution processing on the first input image, and output feature maps (feature map C1, feature map C2, feature map C3, feature map C4) with different resolutions corresponding to the image; that is to say, feature maps of different sizes corresponding to the image are output, and the backbone network completes the extraction of basic features to provide corresponding features for subsequent detection.

[0194] Specifically, the backbone network can perform a series of convolutions on the input image to obtain feature maps at different scales (with different resolutions). These feature maps will provide basic features for subsequent detection modules. The backbone network can take many forms, such as the visual geometry group (VGG), residual neural network (resnet), the core structure of GoogLeNet (Inception-net), etc.

[0195] The backbone network can perform convolution processing on the input image to generate several convolution feature maps of different scales. Each feature map is a H*W*C matrix, where H is the height of the feature map, W is the width of the feature map, and C is the number of channels of the feature map. The backbone can use a variety of existing convolutional network frameworks, such as VGG16, Resnet50, Inception-Net, etc. The following uses Resnet18 as the backbone as an example.

[0196] Assume that the resolution of the input image is H*W*3 (height H, width W, number of channels is 3, that is, RBG three channels). The input image can be convolved through a convolution layer Res18-Conv1 of Resnet18 to generate Featuremap (feature map) C1. This feature map is downsampled twice relative to the input image, and the number of channels is expanded to 64, so the resolution of C1 is H / 4*W / 4*64. C1 can be convolved through Res18-Conv2 of Resnet18 to obtain Featuremap C2, the resolution of this feature map is consistent with C1; C2 continues to be convolved through Res18-Conv3 to generate Featuremap C3, this feature map is further downsampled relative to C2, the number of channels is doubled, and its resolution is H / 8*W / 8*128; finally, C3 is convolved through Res18-Conv4 to generate Featuremap C4, whose resolution is H / 16*W / 16*256.

[0197] It should be noted that the backbone network in the embodiment of the present application may also be referred to as a backbone network, which is not limited here.

[0198] It should be noted that Figure 6 The backbone network shown in the figure is only one implementation method and does not constitute a limitation to the present application.

[0199] In an embodiment of the present application, the network used for target detection may also include one or more parallel heads (or referred to as Headers), each head end head is used to detect a task object in a task (for example, the target object in this embodiment) according to the feature map output by the backbone network, and output a detection frame of the area where the target object is located and a first confidence corresponding to the detection frame; wherein each head in the network used for target detection can complete the detection of different task objects; wherein the task object is an object that needs to be detected in the task; the higher the first confidence, the greater the probability that the target object exists in the detection frame corresponding to the first confidence, or the first confidence is described as indicating the positioning accuracy of the detection frame, and when the positioning accuracy is higher, the greater the probability that the target object exists in the detection frame.

[0200] In the embodiments of the present application, different heads can complete different object detection tasks.

[0201] For example, one head among multiple heads can complete car detection and output the detection box and confidence of Car / Truck / Bus; one head among multiple heads can complete person detection and output the detection box and confidence of Pedestrian / Cyclist / Tricyle; one head among multiple heads can complete traffic light detection and output the detection box and confidence of Red_Trafficligh / Green_Trafficlight / Yellow_TrafficLight / Black_TrafficLight.

[0202] In the embodiment of the present application, the network for object detection may include multiple serial heads; the serial heads are connected to a parallel head; it should be emphasized here that, in fact, the serial head is not necessary, and for scenes where only the detection frame needs to be detected, the serial head is not required. The serial head can perform image segmentation according to the area where the detection frame is located (that is, step 503 in the embodiment).

[0203] Among them, the serial head can be used to: use the detection frame of the task object of the task provided by the connected parallel head to extract the features of the area where the detection frame is located on the feature map, and predict the 3D information, Mask information (or segmentation map) or Keypoint information of the task object of the task according to the features of the area where the detection frame is located. The serial head can complete the 3D / Mask / Keypoint detection of the object inside the detection frame on the basis of detecting the detection frame of the task. For example, the serial 3D_head0 completes the estimation of the vehicle's orientation, center of mass, length, width and height, and outputs the vehicle's 3D frame; the serial Mask_head0 predicts the vehicle's fine mask to segment the vehicle; the serial Keypont_head0 completes the estimation of the vehicle's key points. The serial head is not necessary. Some tasks do not require 3D / Mask / Keypoint detection, so there is no need to connect the serial head in series. For example, the detection of traffic lights only requires the detection frame to be detected, so there is no need to connect the serial head in series. In addition, some tasks can choose to connect one or more serial heads in series according to the specific needs of the task. For example, for parking lot detection, in addition to the detection frame, the key points of the parking space are also required. Therefore, in this task, only one serial Keypoint_head is needed in series, and 3D and Mask heads are not required.

[0204] In the embodiment of the present application, the header can complete the detection of the detection box of a task according to the feature map provided by FPN, output the detection box of the object of this task and the corresponding confidence, etc. The following describes a schematic diagram of the structure of the header, with reference to Figure 7 , Figure 7 It is a kind of header indication, such as Figure 7 As shown in the figure, the head includes three modules: Region Proposal Network (RPN), ROI-ALIGN and RCNN.

[0205] Among them, the RPN module can be used to predict the area where the task object is located on the feature map, and output the candidate detection box that matches the area; or it can be understood that the RPN predicts the area where the task object may exist on the feature map, and gives the boxes of these areas, which are called candidate areas (Proposal). For example, when the head is responsible for detecting cars, its RPN layer predicts the candidate box where the car may exist; when the head is responsible for detecting people, its RPN layer predicts the candidate box where people may exist. Of course, these proposals are inaccurate. On the one hand, they do not necessarily contain the object of the task, and on the other hand, these boxes are not compact.

[0206] The detection candidate region prediction process can be implemented by the head's RPN module, which predicts the area where the task object may exist based on the feature map provided by FPN, and gives the candidate boxes (also called candidate areas, Proposal) of these areas. In this embodiment, if the head is responsible for detecting a car, its RPN layer predicts the candidate boxes where the car may exist.

[0207] The RPN layer can generate the feature map RPN Hidden on the feature map through a 3*3 convolution, for example. The subsequent RPN layer in the head will predict Proposals from RPN Hidden. Specifically, the RPN layer in the head predicts the coordinates and confidence of the Proposals at each position in RPN Hidden through a 1*1 convolution respectively. The higher this confidence, the greater the probability that this Proposal contains an object of this task. For example, the larger the score of a certain Proposal in the head, the greater the probability that it contains a vehicle. The Proposals predicted by each RPN layer need to go through the Proposal merging module to remove redundant Proposals according to the overlap degree between the Proposals (this process can adopt but is not limited to the NMS algorithm), and select the N (N < K) Proposals with the largest scores from the remaining K Proposals as the candidate regions where objects may exist. These Proposals are inaccurate. On the one hand, they may not contain an object of this task. On the other hand, these bounding boxes are not tight. Therefore, the RPN module is only a rough detection process and requires the subsequent RCNN module for further subdivision. When the RPN module regresses the coordinates of the Proposals, it does not directly regress the absolute values of the coordinates, but regresses the coordinates relative to the Anchor. The higher the matching degree between these Anchors and the actual objects, the greater the probability that the RPN can detect the objects.

[0208] The ROI-ALIGN module is used to extract the features of the region where the candidate detection box is located from a feature map provided by the FPN according to the region predicted by the RPN module; that is to say, the ROI-ALIGN module mainly extracts the features of the region where each Proposal is located on a certain feature map according to the Proposals provided by the RPN module, and resizes them to a fixed size to obtain the features of each Proposal. It can be understood that the ROI-ALIGN module can use but is not limited to feature extraction methods such as ROI-POOLING (Region of Interest Pooling) / ROI-ALIGN (Region of Interest Alignment) / PS-ROIPOOLING (Position-Sensitive Region of Interest Pooling) / PS-ROIALIGN (Position-Sensitive Region of Interest Alignment).

[0209] The RCNN module is used to perform convolution processing on the features of the region where the candidate detection frame is located through a neural network to obtain the confidence that the candidate detection frame belongs to each object category; the coordinates of the detection frame of the candidate region are adjusted through a neural network so that the adjusted detection candidate frame is more closely matched to the shape of the actual object than the candidate detection frame, and the adjusted detection candidate frame with a confidence greater than a preset threshold is selected as the detection frame of the region. In other words, the RCNN module mainly refines the features of each Proposal proposed by the ROI-ALIGN module to obtain the confidence that each Proposal belongs to each category (for example, for the task of the car, four scores of Background / Car / Truck / Bus will be given), and at the same time adjusts the coordinates of the detection frame of the Proposal to output a more compact detection frame. These detection frames are merged through non-maximum suppression (NMS) and output as the final detection frame.

[0210] The candidate region classification is mainly composed of Figure 7 The RCNN module of the head in the implementation further regresses the more compact detection frame coordinates according to the features of each Proposal extracted by the ROI-ALIGN module, and classifies the Proposal and outputs the confidence that it belongs to each category. There are many implementable forms of RCNN. The feature size output by the ROI-ALIGN module can be N*14*14*256 (Feature of proposals). In the RCNN module, it is first processed by the convolution module 4 (Res18-Conv5) of Resnet18, and the output feature size is N*7*7*512. Then it is processed by a Global Avg Pool (average pooling layer) to average the 7*7 features in each channel of the input features to obtain N*512 features, where each 1*512-dimensional feature vector represents the features of each Proposal. Next, two fully connected layers FC are used to regress the precise coordinates of the box (output N*4 vectors, where the four values ​​represent the x / y coordinates of the center point of the box, the width and height of the box), and the confidence of the box category (in head0, the score of whether the box is background / car / truck / bus needs to be given). Finally, the box merging operation is used to select several boxes with the largest scores, and the NMS operation is used to remove duplicate boxes, so as to obtain a compact box output.

[0211] In some practical application scenarios, the perception network can also include other heads, which can further perform 3D / Mask / Keypoint detection based on the detection frame. For example, taking 3D as an example, the ROI-ALIGN module extracts the features of the area where each detection frame is located on the feature map output by FPN based on the accurate detection frame provided by the head. Assuming that the number of detection frames is M, the feature size output by the ROI-ALIGN module is M*14*14*256, which is first processed by Resnet18's Res18-Conv5, and the output feature size is N*7*7*512, and then processed by a Global AvgPool (average pooling layer), averaging the 7*7 features of each channel in the input feature to obtain M*512 features, where each 1*512-dimensional feature vector represents the feature of each detection frame. Next, three fully connected layers FC are used to regress the orientation angle (orientation, M*1 vector), centroid coordinates (centroid, M*2 vector, these two values ​​represent the x / y coordinates of the centroid) and length, width and height (dimention) of the object in the frame.

[0212] It should be noted that Figure 7 The header shown in is only one implementation method and does not constitute a limitation of the present application.

[0213] In an embodiment of the present application, target detection may be performed on the first image to obtain a first confidence level of a detection frame of a target object in the first image, and image segmentation may be performed on the image within the detection frame to obtain a segmentation map corresponding to the target object.

[0214] In a possible implementation, the image in the detection frame can be segmented to obtain an initial segmentation map corresponding to the target object, wherein the initial segmentation map includes a plurality of pixels and the probability that each pixel belongs to each category, and the maximum probability value among the probabilities that each pixel belongs to each category is used as the pixel value of each pixel in the segmentation map. Specifically, for a probability segmentation map with an input dimension of (N, C, H, W), N represents the number of images, C represents the number of segmentation categories, and H and W represent the height and width of the image, respectively. The value of each pixel position (n, c, h, w) is between 0 and 1, and the larger the value, the greater the probability that the pixel belongs to category c. For the probability segmentation map (N, C, H, W), firstly obtain the maximum value matrix (N, H, W) and the maximum value subscript matrix (N, H, W) along the second dimension of the matrix, and the maximum value subscript matrix (N, H, W) can be used as a segmentation map.

[0215] In order to calculate the confidence of the segmentation map, the image segmentation quality of the segmentation map can be obtained, and the first confidence of the detection frame can be adjusted based on the image segmentation quality, so that the adjusted first confidence can not only indicate the positioning accuracy of the detection frame but also indicate the image segmentation quality of the segmentation map.

[0216] Next, we describe how to obtain the image segmentation quality of the segmentation map.

[0217] In one implementation, the image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation map, wherein the higher the boundary clarity, the higher the image segmentation quality.

[0218] Taking the target object as a person as an example, the value of each pixel in the segmentation map can be between 0 and 1, wherein the pixel value of the pixel in the background area is 0 or close to 0, and the pixel value of the pixel in the boundary area of ​​the target object is about 0.5. Figure 8 As shown, (a) is the image input, (b) is the probability map, (c) is the segmentation map, (d) is the pixel value less than 0.4, (e) is the point less than 0.6, and (f) is the point less than 0.8. Most of the pixel values ​​inside the human body are 1, or close to 1: Since the 0.5 threshold can be used for binarization when using this segmentation map, errors will occur at the edge. Ideally, the fewer the pixel values ​​of a segmentation map near 0.5 at the edge of the human body, the better the quality of the segmentation map.

[0219] Specifically, the segmentation map includes an object area where the target object is located, a background area of ​​the target object, and a boundary area between the object area and the background area. The area where the pixel points in the segmentation map have pixel values ​​greater than a foreground threshold value are located can be determined as the object area, the area where the pixel points in the segmentation map have pixel values ​​less than a background threshold value are located can be determined as the background area, and the area where the pixel points in the segmentation map have pixel values ​​greater than the background threshold value and less than the foreground threshold value are located can be determined as the boundary area. For example, the background threshold value can be 0.25, then the pixel values ​​of the pixels in the background area are less than 0.25, and the foreground threshold value can be 0.7, then the pixel values ​​of the pixels in the foreground area are greater than 0.7, and then the pixel values ​​of the pixels in the boundary area are greater than 0.25 and less than 0.7.

[0220] In the embodiment of the present application, the ratio of the number of pixels included in the object area to the number of pixels included in the target area can be used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area. In other words, the smaller the boundary area (or the fewer the number of pixels in the boundary area), the higher the image segmentation quality.

[0221] Exemplarily, the image segmentation quality may be calculated as shown in the following formula:

[0222]

[0223] Among them, high_fg_threshold represents the foreground threshold, low_fg_threshold is the background threshold, and a coefficient Pixel_scoring is obtained by calculating the ratio of the number of pixels with pixel values ​​greater than the foreground threshold to the number of pixels greater than the background threshold. The coefficient Pixel_scoring can be used as the image segmentation quality of the segmentation map.

[0224] In another implementation, pixel values ​​of some pixel points included in the segmentation map can be obtained, wherein the pixel values ​​of some pixel points are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1. The image segmentation quality of the segmentation map is determined according to the pixel values ​​of some pixel points included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixel points included in the segmentation map, the higher the image segmentation quality.

[0225] Specifically, the high foreground area can be obtained according to the pixel value in the segmentation map being greater than the threshold value T (that is, the above-mentioned preset value), the mean of the maximum value matrix of the high foreground area can be counted, and a matrix of dimension N is returned, which represents the instance confidence of each picture, and the instance confidence can be used as the image segmentation quality in this embodiment. In addition, the category confidence can also be calculated: traverse all categories C, obtain the area of ​​the category according to the maximum value matrix, obtain the foreground area under the category, and obtain the mean under the category as the confidence of the category, and finally return an N*C matrix.

[0226] In another implementation, initial pixel values ​​of some pixels included in the segmentation map may be obtained, wherein the initial pixel values ​​of some pixels are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1. The initial pixel values ​​are adjusted according to a target mapping relationship to obtain adjusted pixel values ​​of the pixels included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, and in the target mapping relationship, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases, and the image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map, the higher the image segmentation quality.

[0227] In this embodiment, the pixel values ​​of the pixels in the segmentation map are mapped based on a preset mapping relationship, wherein the image segmentation quality of the segmentation map is expressed based on the average value of the mapped pixel values, which is equivalent to using the mapped pixel values ​​as the weights when calculating the image segmentation quality of the pixels. The larger the weight, the greater the effect of improving the image segmentation quality, and the smaller the weight, the greater the effect of reducing the image segmentation quality. Specifically, for the pixel points whose pixel values ​​are less than the foreground threshold and greater than or equal to the background threshold, they are the cause of the low image segmentation quality of the segmentation map, so the weight assigned to this part of the pixels is very small, for example, a value of 0 or close to 0 is assigned. Therefore, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases, that is, at the initial pixel value close to the preset value, the assigned weight is close to 0, and as the initial pixel value increases, the assigned weight gradually increases. When the initial pixel value is 1, the assigned weight reaches 1.

[0228] In this embodiment, as the initial pixel value increases, the slope of the target mapping relationship may gradually increase, so the target mapping relationship is equivalent to a mapping relationship that becomes increasingly steeper.

[0229] In this embodiment, the target mapping relationship may be a preset function relationship, for example, Fig. 9 , Fig. 9 This is a schematic diagram of various target mapping relationships in this embodiment, where Fig. 9 The horizontal axis is the initial pixel value (where the minimum initial pixel value is greater than the preset value, for example Fig. 9 The default value is 0.5), where Fig. 9The vertical axis is a value from 0 to 1. Fig. 9 In the two lower functional relationships among the various functional relationships shown, as the initial pixel value increases, the slope of the target mapping relationship increases. Fig. 9 In the topmost functional relationship among the multiple functional relationships shown, as the initial pixel value increases, the slope of the target mapping relationship decreases. Fig. 9 In the second functional relationship from the top among the multiple functional relationships shown, as the initial pixel value increases, the slope of the target mapping relationship remains unchanged.

[0230] In an embodiment of the present application, the image segmentation quality of the segmentation map can be determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map, the higher the image segmentation quality.

[0231] Exemplarily, the image segmentation quality of the segmentation map can be calculated by the following formula:

[0232]

[0233] Where F is the target mapping relationship, which maps pixel points with pixel values ​​greater than 0.5 to values ​​in another interval of [0.5-1.0]. The important point is that values ​​closer to 1 are given higher mapping values. The numerator is the sum of the adjusted pixel values ​​of the pixels included in the segmentation map, and the denominator is the number of pixels whose initial pixel values ​​are greater than a preset value (for example, 0.5).

[0234] 504. Adjust the first confidence level according to the image segmentation quality to obtain a second confidence level of the segmentation map, wherein the higher the image segmentation quality is, the greater the second confidence level is.

[0235] In the embodiment of the present application, after the image segmentation quality of the segmentation map is obtained, the first confidence level may be adjusted based on the image segmentation quality to determine a second confidence level as the segmentation map.

[0236] In one implementation, the product of the image segmentation quality and the first confidence level may be determined as the second confidence level of the segmentation map.

[0237] Through the above method, the second confidence of the segmentation map can indicate both the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, wherein the higher the positioning accuracy, the higher the second confidence, and the higher the positioning accuracy of the detection box, the higher the second confidence.

[0238] In one implementation, the terminal device may display an AR object around the target object in the first image based on the second confidence level being higher than a threshold.

[0239] In an embodiment of the present application, in a scenario where AR special effects are displayed, the terminal device can display an AR object around the target object in the first image based on the second confidence level being higher than a threshold. Specifically, the AR object can be displayed around the target object based on the 3D position information of the target object.

[0240] In one implementation, the terminal device may replace the target object in the first image with a first object based on the second confidence being higher than a threshold, where the first object is different from the target object.

[0241] In an embodiment of the present application, in the scenario of AI removing pedestrians, if the target object is a determined moving pedestrian, the terminal device can replace the target object with a first object based on the second confidence level being higher than a threshold, wherein the first object is the background area where the target object is located in other image frames (that is, in the first image, area A is the area where the target object is located, and in other image frames, area A is the background area where the target object is located, and the target object has moved to other locations in the image or is not in the image).

[0242] Next, an application example provided by this embodiment is described by taking AR special effect display as an example:

[0243] First, the neural network for target detection is trained: first, the portrait segmentation data (I) and the portrait 3D information data (V) are labeled, the data set is obtained and the joint network is trained, the model output result is compared with the real label, the difference is calculated, and the network parameters are updated inversely using the difference until the pre-set number of training times is reached; during the model inference process, the RGB video stream is input into the joint model network, and for each frame of the image, the 3D information and segmentation mask (or segmentation map) of the frame are obtained; it is determined whether the frame needs to display the AR wing effect. Specifically, it can be used Figure 5 The segmentation map confidence determination method in the corresponding embodiment estimates a confidence for the segmentation mask. If the confidence is greater than 0.6, the frame outputs the AR wing effect. If it is determined to output the AR wing effect, the wing position is reconstructed using 3D information, and then the segmentation mask is used for virtual and real occlusion.

[0244] The embodiment of the present application provides a method for determining the confidence of a segmentation map, the method comprising: acquiring a first image; performing target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence of the detection frame, the first confidence being used to indicate the positioning accuracy of the detection frame; performing image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and to obtain the image segmentation quality of the segmentation map; adjusting the first confidence according to the image segmentation quality to obtain a second confidence of the segmentation map, wherein the higher the image segmentation quality, the greater the second confidence. In the above manner, the first confidence of the detection frame is adjusted using the image segmentation quality of the segmentation map to obtain the second confidence of the segmentation map, thereby making the second confidence include not only the positioning accuracy of the detection frame, but also the segmentation quality information of the segmentation map itself, so as to obtain a segmentation map confidence with higher accuracy. Moreover, the segmentation map confidence determination method in this embodiment does not require the addition of an additional network to calculate the confidence of the segmentation map. The accurate segmentation map confidence can be directly obtained based on the results of target detection and the results of the segmentation map. There is no need to add a lot of additional calculations, which is more friendly to the terminal side, making the solution easy to deploy on the terminal side for implementation.

[0245] Reference Fig.10 , Fig.10 A schematic diagram of the structure of a segmentation image confidence determination device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the segmentation map confidence determination device 1000 provided in the embodiment of the present application includes:

[0246] An acquisition module 1001 is used to acquire a first image;

[0247] For the specific description of the acquisition module 1001, please refer to the description of step 501, which will not be repeated here.

[0248] The target detection module 1002 is used to perform target detection on the first image to obtain a detection frame of the target object in the first image and a first confidence of the detection frame, where the first confidence is used to indicate the positioning accuracy of the detection frame;

[0249] For a detailed description of the target detection module 1002 , please refer to the description of step 502 , which will not be repeated here.

[0250] An image segmentation module 1003 is used to perform image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and obtain image segmentation quality of the segmentation map;

[0251] For the detailed description of the image segmentation module 1003 , please refer to the description of step 503 , which will not be repeated here.

[0252] The confidence determination module 1004 is used to adjust the first confidence according to the image segmentation quality to obtain a second confidence of the segmentation map, wherein the higher the image segmentation quality, the greater the second confidence.

[0253] For a detailed description of the confidence determination module 1004 , please refer to the description of step 504 , which will not be repeated here.

[0254] In a possible implementation, the image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation image, wherein the higher the boundary clarity, the higher the image segmentation quality.

[0255] In a possible implementation, the second confidence of the segmentation map is used to indicate the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, wherein the higher the positioning accuracy is, the higher the second confidence is.

[0256] In a possible implementation, the segmentation map includes an object region where the target object is located, a background region of the target object, and a boundary region between the object region and the background region, and the acquisition module is used to:

[0257] The ratio of the number of pixels included in the object area to the number of pixels included in the target area is used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area.

[0258] In a possible implementation, the acquisition module is used to:

[0259] Determine the area where the pixels in the segmentation map whose pixel values ​​are greater than the foreground threshold are located as the object area;

[0260] Determine the area where the pixels in the segmentation map whose pixel values ​​are less than the background threshold are located as the background area;

[0261] The region where the pixel points in the segmentation image whose pixel values ​​are greater than the background threshold and less than the foreground threshold are located is determined as the boundary region.

[0262] In a possible implementation, the image segmentation module is used to perform image segmentation on the image in the detection frame to obtain an initial segmentation map corresponding to the target object, wherein the initial segmentation map includes a plurality of pixels and a probability that each pixel belongs to each category;

[0263] The maximum probability value among the probabilities that each pixel point belongs to each category is used as the pixel value of each pixel point in the segmentation map;

[0264] Obtaining pixel values ​​of some pixels included in the segmentation map, wherein the pixel values ​​of some pixels are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1;

[0265] The image segmentation quality of the segmentation map is determined according to the pixel values ​​of some pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

[0266] In a possible implementation, the image segmentation module is used to obtain initial pixel values ​​of some pixel points included in the segmentation map, wherein the initial pixel values ​​of some pixel points are greater than a preset value, the preset value is less than the foreground threshold and greater than or equal to the background threshold, and the foreground threshold is less than 1;

[0267] According to the target mapping relationship, the initial pixel value is adjusted to obtain an adjusted pixel value of the pixel point included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, and in the target mapping relationship, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases;

[0268] The image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

[0269] In a possible implementation, the confidence determination module is used to determine a product of the image segmentation quality and the first confidence as a second confidence of the segmentation map.

[0270] In a possible implementation, the device further includes:

[0271] an image processing module, configured to display an AR object around the target object in the first image based on the second confidence level being higher than a threshold; or,

[0272] Based on the second confidence being higher than a threshold, the target object in the first image is replaced with a first object, the first object being different from the target object.

[0273] The embodiment of the present application provides a segmentation map confidence determination device, the device comprising: an acquisition module for acquiring a first image; a target detection module for performing target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence of the detection frame, the first confidence being used to indicate the positioning accuracy of the detection frame; an image segmentation module for performing image segmentation on the image in the detection frame to obtain a segmentation map corresponding to the target object and to obtain the image segmentation quality of the segmentation map; a confidence determination module for adjusting the first confidence according to the image segmentation quality to obtain a second confidence of the segmentation map, wherein the higher the image segmentation quality, the greater the second confidence. In the above manner, the first confidence of the detection frame is adjusted using the image segmentation quality of the segmentation map to obtain the second confidence of the segmentation map, so that the second confidence includes not only the positioning accuracy of the detection frame, but also the segmentation quality information of the segmentation map itself, so that a segmentation map confidence with higher accuracy can be obtained. Moreover, the segmentation map confidence determination method in this embodiment does not require the addition of an additional network to calculate the confidence of the segmentation map. The accurate segmentation map confidence can be directly obtained based on the results of target detection and the results of the segmentation map. There is no need to add a lot of additional calculations, which is more friendly to the terminal side, making the solution easy to deploy on the terminal side for implementation.

[0274] The present application also provides a server. Fig.11 , Fig.11 1 is a schematic diagram of the structure of a server provided in an embodiment of the present application. Specifically, the server 1100 is implemented by one or more servers. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1111 (for example, one or more processors) and memory 1132, and one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or persistent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1111 can be configured to communicate with the storage medium 1130 and execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0275] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158; or, one or more operating systems 1141, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0276] Specifically, the server can execute Figure 5 The segmentation map confidence determination method in the corresponding embodiment.

[0277] Fig.10 The segmentation map confidence determination apparatus 1000 described in the specification may be a module in a server, and a processor in the server may execute the segmentation map confidence determination method executed by the segmentation map confidence determination apparatus 1000 .

[0278] The embodiment of the present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned server.

[0279] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0280] The server or terminal device provided in the embodiment of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0281] For details, please refer to Fig.12 , Fig.12A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 1200, NPU 1200 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 1203, which is controlled by the controller 1204 to extract matrix data from the memory and perform multiplication operations.

[0282] NPU 1200 can achieve Figure 6 The model training method provided in the described embodiments, or reasoning on the trained model.

[0283] Among them, the computing circuit 1203 in the NPU 1200 can execute the steps of obtaining a first neural network model and performing model training on the first neural network model.

[0284] More specifically, in some implementations, the operation circuit 1203 in the NPU 1200 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1203 is a two-dimensional systolic array. The operation circuit 1203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1203 is a general-purpose matrix processor.

[0285] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1202 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1201 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1208.

[0286] The unified memory 1206 is used to store input data and output data. The weight data is directly transferred to the weight memory 1202 through the direct memory access controller (DMAC) 1205. The input data is also transferred to the unified memory 1206 through the DMAC.

[0287] BIU stands for Bus Interface Unit, i.e., bus interface unit 1210 , which is used for the interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 1209 .

[0288] The bus interface unit 1210 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1209 to obtain instructions from the external memory, and is also used for the storage unit access controller 1205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0289] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1206 or to transfer weight data to the weight memory 1202 or to transfer input data to the input memory 1201.

[0290] The vector calculation unit 1207 includes multiple operation processing units, and further processes the output of the operation circuit 1203 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0291] In some implementations, the vector calculation unit 1207 can store the processed output vector to the unified memory 1206. For example, the vector calculation unit 1207 can apply a linear function; or a nonlinear function to the output of the operation circuit 1203, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 1207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1203, for example, for use in a subsequent layer in a neural network.

[0292] An instruction fetch buffer 1209 connected to the controller 1204 is used to store instructions used by the controller 1204;

[0293] Unified memory 1206, input memory 1201, weight memory 1202 and instruction fetch memory 1209 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0294] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0295] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0296] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0297] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0298] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for determining the confidence of a segmentation map, characterized in that: The method comprises: acquiring a first image; Performing target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence of the detection frame, where the first confidence is used to indicate a positioning accuracy of the detection frame; Performing image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and to obtain image segmentation quality of the segmentation map; According to the image segmentation quality, the first confidence is adjusted to obtain a second confidence of the segmentation map, wherein the second confidence of the segmentation map is used to indicate the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, and the higher the positioning accuracy, the higher the second confidence, and the higher the image segmentation quality, the greater the second confidence.

2. The method according to claim 1, characterized in that The image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation image, wherein the higher the boundary clarity is, the higher the image segmentation quality is.

3. The method according to claim 1, characterized in that The method further comprises: Determine the area where the pixels in the segmentation map whose pixel values ​​are greater than the foreground threshold are located as the object area; Determine the area where the pixels in the segmentation map whose pixel values ​​are less than the background threshold are located as the background area; Determine the region where the pixel points in the segmentation image whose pixel values ​​are greater than the background threshold and less than the foreground threshold are located as the boundary region; The obtaining the image segmentation quality of the segmentation map includes: The ratio of the number of pixels included in the object area to the number of pixels included in the target area is used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area.

4. The method according to any one of claims 1 to 3, characterized in that: The performing image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object includes: Performing image segmentation on the image within the detection frame to obtain an initial segmentation map corresponding to the target object, wherein the initial segmentation map includes a plurality of pixels and a probability that each pixel belongs to each category; The maximum probability value among the probabilities that each pixel point belongs to each category is used as the pixel value of each pixel point in the segmentation map; The obtaining the image segmentation quality of the segmentation map includes: Obtaining pixel values ​​of some pixel points included in the segmentation map, wherein the pixel values ​​of some pixel points are greater than a preset value, the preset value is less than a foreground threshold and greater than or equal to a background threshold, and the foreground threshold is less than 1; The image segmentation quality of the segmentation map is determined according to the pixel values ​​of some pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

5. The method according to any one of claims 1 to 3, characterized in that: The obtaining the image segmentation quality of the segmentation map includes: Acquire initial pixel values ​​of some pixels included in the segmentation map, wherein the initial pixel values ​​of some pixels are greater than a preset value, the preset value is less than a foreground threshold and greater than or equal to a background threshold, and the foreground threshold is less than 1; According to the target mapping relationship, the initial pixel value is adjusted to obtain an adjusted pixel value of the pixel point included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, and in the target mapping relationship, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases; The image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

6. The method according to any one of claims 1 to 3, characterized in that: The adjusting the first confidence level according to the image segmentation quality to obtain a second confidence level of the segmentation map includes: A product of the image segmentation quality and the first confidence is determined as a second confidence of the segmentation map.

7. The method according to any one of claims 1 to 3, characterized in that: After obtaining the second confidence of the segmentation map, the method further includes: Based on the second confidence being higher than a threshold, displaying an AR object around the target object in the first image; or, Based on the second confidence being higher than a threshold, the target object in the first image is replaced with a first object, the first object being different from the target object.

8. A device for determining confidence of a segmentation image, characterized in that: The device comprises: An acquisition module, used for acquiring a first image; a target detection module, configured to perform target detection on the first image to obtain a detection frame of a target object in the first image and a first confidence level of the detection frame, wherein the first confidence level is used to indicate a positioning accuracy of the detection frame; An image segmentation module is used to perform image segmentation on the image within the detection frame to obtain a segmentation map corresponding to the target object and to obtain the image segmentation quality of the segmentation map; A confidence determination module is used to adjust the first confidence according to the image segmentation quality to obtain a second confidence of the segmentation map, wherein the second confidence of the segmentation map is used to indicate the positioning accuracy of the detection box and the image segmentation quality of the segmentation map, and the higher the positioning accuracy, the higher the second confidence, and the higher the image segmentation quality, the greater the second confidence.

9. The device according to claim 8, characterized in that The image segmentation quality is used to indicate the boundary clarity of the target object in the segmentation image, wherein the higher the boundary clarity is, the higher the image segmentation quality is.

10. The device according to any one of claims 8 to 9, characterized in that: The acquisition module is used to: Determine the area where the pixels in the segmentation map whose pixel values ​​are greater than the foreground threshold are located as the object area; Determine the area where the pixels in the segmentation map whose pixel values ​​are less than the background threshold are located as the background area; Determine the region where the pixel points in the segmentation image whose pixel values ​​are greater than the background threshold and less than the foreground threshold are located as the boundary region; The ratio of the number of pixels included in the object area to the number of pixels included in the target area is used as the image segmentation quality of the segmentation map, wherein the target area is the union of the object area and the boundary area.

11. The device according to any one of claims 8 to 9, characterized in that: The image segmentation module is used to perform image segmentation on the image in the detection frame to obtain an initial segmentation map corresponding to the target object, wherein the initial segmentation map includes a plurality of pixels and a probability that each pixel belongs to each category; The maximum probability value among the probabilities that each pixel point belongs to each category is used as the pixel value of each pixel point in the segmentation map; Obtaining pixel values ​​of some pixel points included in the segmentation map, wherein the pixel values ​​of some pixel points are greater than a preset value, the preset value is less than a foreground threshold and greater than or equal to a background threshold, and the foreground threshold is less than 1; The image segmentation quality of the segmentation map is determined according to the pixel values ​​of some pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

12. The device according to any one of claims 8 to 9, characterized in that: The image segmentation module is used to obtain initial pixel values ​​of some pixels included in the segmentation map, wherein the initial pixel values ​​of some pixels are greater than a preset value, the preset value is less than a foreground threshold and greater than or equal to a background threshold, and the foreground threshold is less than 1; According to the target mapping relationship, the initial pixel value is adjusted to obtain an adjusted pixel value of the pixel point included in the segmentation map, wherein the target mapping relationship represents a mapping relationship between the initial pixel value and the adjusted pixel value, and in the target mapping relationship, as the initial pixel value changes from the preset value to 1, the adjusted pixel value gradually changes from 0 to 1, and as the initial pixel value increases, the slope of the target mapping relationship remains unchanged or increases; The image segmentation quality of the segmentation map is determined according to the adjusted pixel values ​​of the pixels included in the segmentation map, wherein the larger the average value of the adjusted pixel values ​​of the pixels included in the segmentation map is, the higher the image segmentation quality is.

13. The device according to any one of claims 8 to 9, characterized in that: The confidence determination module is used to determine the product of the image segmentation quality and the first confidence as the second confidence of the segmentation map.

14. The device according to any one of claims 8 to 9, characterized in that: The device also includes: an image processing module, configured to display an AR object around the target object in the first image based on the second confidence level being higher than a threshold; or, Based on the second confidence being higher than a threshold, the target object in the first image is replaced with a first object, the first object being different from the target object.

15. A model training device, characterized in that: The device includes a memory and a processor; the memory stores codes, and the processor is configured to obtain the codes and execute the method according to any one of claims 1 to 7.

16. A computer storage medium, characterized in that: The computer storage medium stores one or more instructions, which, when executed by one or more computers, enable the one or more computers to implement the method of any one of claims 1 to 7.

17. A computer program product comprising code, characterized in that When the code is executed, it is used to implement the method according to any one of claims 1 to 7.

18. A chip system, characterized in that: It comprises a processor, wherein the processor is used to execute processing according to the acquired computer instructions, so as to implement the method according to any one of claims 1 to 7.