Image processing method and device, equipment and storage medium

Abstract

The invention discloses an image processing method and device, equipment and a storage medium, and relates to the technical field of computers. The method comprises the following steps: detecting a first image frame of a first video to obtain a contour line of a first object contained in the first image frame; n contour points on the contour line are extracted, contour point data corresponding to the first image frame are obtained, the contour point data are used for indicating the positions of the N contour points in the first image frame, and N is a positive integer; and sending the first image frame and the contour point data corresponding to the first image frame to the first device. According to the method, the steps of detecting the first object and extracting the contour points are transferred to the second equipment, so that relatively high calculation pressure on the first equipment is avoided, and calculation resources consumed by the first equipment are reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art

[0002] For live video, users can comment on the video by sending barrages. The barrages are usually located above the video layer. In order to avoid the barrages blocking the characters in the video, the barrages need to be cut out according to the outline of the characters.

[0003] The barrage cutout used in related technologies is to detect the character area in the video in real time on the audience-end device corresponding to the live video, and segment the character area and the video, and then draw the segmented character area into a transparent image to obtain the character image, and then draw the character image on the top layer of the video image in real time, thereby achieving the display effect of the barrage being located at the bottom layer of the character image.

[0004] However, the above solution requires person detection and person segmentation to be performed on the viewer-side device, which consumes more computing resources of the viewer-side device. Summary of the invention

[0005] The present application provides an image processing method, apparatus, device and storage medium. The technical solution is as follows:

[0006] According to one aspect of an embodiment of the present application, there is provided an image processing method, the method comprising:

[0007] Detecting a first image frame of a first video to obtain a contour line of a first object contained in the first image frame;

[0008] Extracting N contour points on the contour line to obtain contour point data corresponding to the first image frame, wherein the contour point data is used to indicate positions of the N contour points in the first image frame, where N is a positive integer;

[0009] The first image frame and contour point data corresponding to the first image frame are sent to a first device.

[0010] According to one aspect of an embodiment of the present application, there is provided an image processing method, the method comprising:

[0011] Receive a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame; wherein the contour point data is used to indicate positions of N contour points on a contour line of a first object contained in the first image frame in the first image frame, where N is a positive integer;

[0012] determining a contour line of the first object contained in the first image frame according to the contour point data;

[0013] The first image frame is displayed, and associated information of the first image frame obtained based on the contour line of the first object is displayed.

[0014] According to one aspect of an embodiment of the present application, there is provided an image processing device, the device comprising:

[0015] A detection module, configured to detect a first image frame of a first video to obtain a contour line of a first object contained in the first image frame;

[0016] An extraction module, used to extract N contour points on the contour line to obtain contour point data corresponding to the first image frame, wherein the contour point data is used to indicate positions of the N contour points in the first image frame, where N is a positive integer;

[0017] A sending module is used to send the first image frame and the contour point data corresponding to the first image frame to the first device.

[0018] In some embodiments, the extraction module includes: an extraction unit, configured to extract M contour points on the contour line, where M is a positive integer greater than N. A sampling unit, configured to sample the N contour points from the M contour points according to a sampling step length to obtain contour point data corresponding to the first image frame.

[0019] In some embodiments, the extraction unit is used to obtain a contour image corresponding to the first image frame, the contour image being an image in which the inside and outside of the contour line in the first image frame are represented by different colors, the inside of the contour line being represented by a first color, and the outside of the contour line being represented by a second color; for each row of pixel points in the contour image, two pixel points at the connection position of the first color and the second color are determined as two contour points; and the M contour points are obtained based on the two contour points respectively determined from each row of pixel points in the contour image.

[0020] In some embodiments, the sampling unit is further used to calculate the sampling step size according to the number M of contour points on the contour line and the number N of contour points to be extracted; wherein N is a preset fixed value.

[0021] In some embodiments, the device also includes an image frame extraction module, which is used to extract an image frame from the first video every T image frames as the first image frame, where T is a value that dynamically changes based on the content of the first video, and T is a positive integer.

[0022] In some embodiments, the device also includes an adjustment module, which is used to obtain at least one pair of contour points located at the same position in the sequence from the first contour point sequence and the second contour point sequence for two adjacent first image frames extracted; wherein the first contour point sequence is a contour point sequence corresponding to one of the two adjacent first image frames, and the second contour point sequence is a contour point sequence corresponding to the other of the two adjacent first image frames, and the contour point sequence is a sequence obtained by arranging the N contour points extracted from the first image frames in a set order; calculate the distance value between the two contour points contained in each pair of contour points, and the distance value is used to characterize the difference in the positions of the two contour points in the first image frame; and adjust T according to the distance value.

[0023] In some embodiments, the adjustment module is used to reduce the value of T if the number of distance values ​​greater than a first threshold satisfies the first number; increase the value of T if the number of distance values ​​less than a second threshold satisfies the second number; and keep the value of T unchanged if the number of distance values ​​greater than the first threshold does not satisfy the first number, and the number of distance values ​​less than the second threshold does not satisfy the second number.

[0024] In some embodiments, the apparatus further includes an encoding module, which is used to encode the first image frame to obtain first code stream data; encapsulate the contour point data corresponding to the first image frame into supplemental enhancement information SEI, and encode the SEI to obtain second code stream data; and send a video code stream of the first video to the first device, wherein the video code stream includes the first code stream data and the second code stream data.

[0025] In some embodiments, the contour point data corresponding to the first image frame includes: the position coordinates of a first contour point among the N contour points, and the position offsets of other contour points among the N contour points except the first contour point relative to the first contour point.

[0026] In some embodiments, the first video is a live video, the method is executed by a second device, the second device is a host-end device corresponding to the live video, and the first device is a viewer-end device corresponding to the live video.

[0027] According to one aspect of an embodiment of the present application, there is provided an image processing device, the device comprising:

[0028] A receiving module, configured to receive a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame; wherein the contour point data is used to indicate positions of N contour points on a contour line of a first object contained in the first image frame in the first image frame, where N is a positive integer;

[0029] a determination module, configured to determine a contour line of the first object contained in the first image frame according to the contour point data;

[0030] The display module is configured to display the first image frame and display associated information of the first image frame obtained based on the contour line of the first object.

[0031] In some embodiments, the determination module is used to perform interpolation processing on the N contour points according to the contour point data to obtain P contour points, where P is a positive integer greater than N; and determine the contour line of the first object contained in the first image frame according to the P contour points.

[0032] In some embodiments, the receiving module is used to receive the video code stream of the first video sent by the second device, the video code stream including first code stream data and second code stream data; decode the first code stream data to obtain the first image frame; decode the second code stream data to obtain supplemental enhancement information SEI, and obtain contour point data corresponding to the first image frame from the SEI.

[0033] In some embodiments, an image frame is extracted from every T image frames in the first video as the first image frame, where T is a positive integer; the display module is used to execute the step of determining the contour line of the first object contained in the first image frame according to the contour point data if the image frame to be displayed is the first image frame; if the image frame to be displayed is not the first image frame, determining the contour line of the first object contained in the most recent first image frame before the image frame to be displayed as the contour line of the first object contained in the image frame to be displayed.

[0034] In some embodiments, the associated information of the first image frame includes bullet chat information of the first image frame; the display module is used to display the first image frame, and display the bullet chat information of the first image frame outside the contour line of the first object.

[0035] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned image processing method.

[0036] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned image processing method.

[0037] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. The computer program is loaded and executed by a processor to implement the above-mentioned image processing method.

[0038] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0039] By detecting the first image frame of the first video on the second device (such as the anchor terminal device or the server), the contour line of the first object contained in the first image frame and the N contour points on the contour line are obtained, and after the contour point data corresponding to the first image frame and the N contour points are sent to the first device (such as the audience terminal device), the first device can determine the contour area of ​​the first object in the first image frame according to the first image frame and the corresponding contour point data. Compared with the related art in which the first device detects the first image frame of the first video, the technical solution provided by the present application transfers the steps of detecting the first object and extracting contour points to the second device, avoiding a large computing pressure on the first device, reducing the computing resources required by the first device, and improving the operating efficiency of the first device. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0041] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0042] Figure 3 is a schematic diagram of a contour line of a first object provided by an embodiment of the present application;

[0043] Figure 4 is a flow chart of a method for acquiring contour point data provided by an embodiment of the present application;

[0044] Figure 5 is a schematic diagram of a contour image corresponding to a first image frame provided by an embodiment of the present application;

[0045] Figure 6 is a schematic diagram of contour points in a first image frame provided by an embodiment of the present application;

[0046] Figure 7 is a flow chart of adjusting the T value provided by an embodiment of the present application;

[0047] Figure 8 is a flowchart of an image processing method provided by another embodiment of the present application;

[0048] Fig. 9 This is a schematic diagram of the display effect after the bullet screen is cropped provided by an embodiment of the present application;

[0049] Fig.10 It is a specific implementation process diagram of the image processing method provided by an embodiment of the present application;

[0050] Fig.11 is a flowchart of an image processing method after the second device and the first device are combined, provided by an embodiment of the present application;

[0051] Fig.12 is a block diagram of an image processing device provided by an embodiment of the present application;

[0052] Fig.13 is a block diagram of an image processing device provided by another embodiment of the present application;

[0053] Fig.14 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0055] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0056] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0057] Computer vision (CV) is a science that studies how to make machines "see". To be more specific, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the field of vision such as swin-transformer, ViT (Vision Transformers), V-MOE (Vision Mixture of Expert), and MAE (Masked AutoEncoder) can be quickly and widely applied to downstream specific tasks after fine tuning. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (three Dimensions) technology, virtual reality, augmented reality, simultaneous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0058] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (Artificial Intelligence Generated Content, AIGC), conversational interaction, smart medical care, smart customer service, game AI, virtual reality (Virtual Reality, VR), augmented reality (Augmented Reality, AR), etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0059] The technical solution of this application mainly involves computer vision technology and voice technology in artificial intelligence technology, and mainly involves the image processing process when the anchor-end device is broadcasting live.

[0060] Please refer to Figure 1, which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as image processing. The solution implementation environment may include: a first device 10, a second device 20 and a server 30.

[0061] The first device 10 refers to a terminal device used by a first user, and the second device 20 refers to a terminal device used by a second user. The first device 10 and the second device 20 can be electronic devices such as mobile phones, tablet computers, laptop computers, multimedia playback devices, PCs (Personal Computers), and vehicle-mounted terminals, but are not limited thereto.

[0062] In some embodiments, a client of a target application is installed in the first device 10, and the target application has a function of displaying a video, and the first user can watch the first video through the target application. Exemplarily, the target application can be a web browsing program (or browser), a social application, an instant messaging application, a video application, a live broadcast application, a game application, etc., which is not limited in this application. The first device 10 displays a first video, and optionally, the first video can be a live video or a non-live video.

[0063] In the embodiment of the present application, taking the first video as a live video as an example, the first video can be provided by the second device 20. A live broadcast application is installed in the second device 20, and the live broadcast application has the function of performing real-time online video live broadcast, and the second user can perform video live broadcast through the live broadcast application. Exemplarily, the live broadcast video (i.e., the first video) of the live broadcast application can be a live broadcast video of a person of the second user, or a live broadcast video of a scene operated by the second user, or a real-time game screen of the second user in a game application, or a real-time teaching screen of the second user in a learning application, and the present application does not limit this.

[0064] Through the live video function, the live screen of the second user in the live broadcast application can be used to generate a corresponding live video (i.e., the first video), and then the live video can be sent to the server 30. The server 30 sends the above live video to the first device 10. After the first device 10 receives the live video, it displays it in the target application.

[0065] The live broadcast application installed and running in the second device 20 may be the same application as the target application installed and running in the first device 10, or may be another application different from the target application. If the live broadcast application and the target application are the same application, the application has the function of acquiring the live broadcast video in the client of the second user and sending the live broadcast video to the backend server of the client of the first user.

[0066] There can be one or more first devices 10 , and there can be one or more second devices 20 . A live video in the second device 20 can be sent to one or more clients of the first device 10 .

[0067] The server 30 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The first device 10 and the second device 20 may communicate with the server 30 respectively, and the server 30 is used to provide background services for the clients of the applications installed and running in the first device 10 and the second device 20. For example, the server 30 may be a background server of the target application, and optionally, the server may provide background services for the clients installed and running in multiple first devices 10 at the same time. It can obtain live video from the second device 20 and send the live video to the first device 10.

[0068] Please refer to Figure 2 , which shows a flow chart of an image processing method provided by an embodiment of the present application. The execution subject of each step of the method may be a second device. The method may include at least one of the following steps 210 to 230:

[0069] Step 210: Detect a first image frame of a first video to obtain a contour line of a first object contained in the first image frame.

[0070] The first video refers to any video, and the first image frame refers to any image frame in the first video. Optionally, the first image frame may include the first object or may not include the first object.

[0071] The first object includes, but is not limited to, objects such as people, animals, plants, and objects, and may be real objects or virtual objects. For example, if the first object refers to a person, the first object may be a real person, for example, a well-known person such as an athlete, a scientist, an entrepreneur, or an ordinary person such as a colleague, a neighbor, or a stranger; the first object may also be a virtual person, and when the first object is a virtual person, the first object may not be limited to human form, and may include any virtual form created independently, for example, it may be a character appearing in a film or TV series, or a character created independently in an animation work or a game work, such as virtual characters such as "Pixu" and "Doraemon".

[0072] The contour line of the first object refers to a dividing line between the first object and the background image in the first image frame, and is used to separate the first object from the background image, wherein the background image refers to an image other than the first object in the first image frame.

[0073] Performing image detection on the first image frame of the first video, first determining whether the first image frame contains the first object, and if the first image frame contains the first object, extracting the contour of the first object. Figure 3 As shown, the first image frame 301 includes a contour line 302 of a first object, and the contour line 302 of the first object is an object contour line obtained after segmenting the first object and the background image.

[0074] In some embodiments, the first video is a live video. Figure 2 The image processing method shown is executed by the second device, the second device is the anchor end device corresponding to the live video, and the first device is the audience end device corresponding to the live video.

[0075] Live video has been introduced in the above embodiment and will not be repeated here. In the case where the first video is a live video, the second device generates each image frame of the first video in real time by installing and running a live application, and obtains the contour point data corresponding to the first image frame in real time, and then sends the image frame of the first video and the corresponding contour point data to the server, and the server sends the image frame of the first video and the corresponding contour point data to the first device, and the first device receives the image frame and the corresponding contour point data in real time by installing and running the target application, and watches it.

[0076] The anchor-end device mentioned here refers to the terminal device that generates live video, and the audience-end device refers to the terminal device that receives and watches live video.

[0077] Optionally, Figure 2The image processing method shown can also be executed by a server, which provides background services for the live broadcast application installed and running in the second device. The server obtains the live broadcast data on the client of the second device in real time, integrates and processes the live broadcast data, generates each image frame and corresponding contour point data, and then sends each image frame and corresponding contour point data to the first device and the second device respectively, so that the second user can observe the current live broadcast situation in real time on the second device, and the first user can watch the live broadcast video in real time through the client on the first device.

[0078] By real-time monitoring of live video and processing of image frames, it is not limited to the processing of existing videos, which expands the application scenarios of the image generation method proposed in this application and improves the breadth of application of the method of this application.

[0079] Step 220 , extracting N contour points on the contour line to obtain contour point data corresponding to the first image frame, the contour point data is used to indicate the positions of the N contour points in the first image frame, and N is a positive integer.

[0080] The contour points refer to the pixel points on the contour line, and the contour point data refer to the position data of the N contour points, which are used to indicate the positions of the N contour points in the first image frame. Optionally, the N contour points can be N pixel points evenly distributed on the contour line, or can be N pixel points randomly extracted from the contour line.

[0081] In some embodiments, the contour point data corresponding to the first image frame includes: the position coordinates of the first contour point among the N contour points, and the position offsets of other contour points among the N contour points except the first contour point relative to the first contour point.

[0082] The first contour point refers to any one of the N contour points. For example, the contour point located at the lower left of the first image frame among the N contour points can be taken as the first contour point.

[0083] Optionally, a coordinate system can be established with the position of the first contour point among the N contour points as the origin to obtain the position coordinates of other contour points among the N contour points except the first contour point. Thus, the position coordinates of the first contour point and the position coordinates of other contour points except the first contour point can be determined as contour point data corresponding to the first image frame.

[0084] Optionally, the position of the lower left corner of the first image frame can be used as the origin to establish a coordinate system, thereby obtaining the position coordinates of each contour point in the N contour points, and the position offset of other contour points other than the first contour point relative to the first contour point can be calculated based on the position coordinates of the first contour point. Therefore, the position coordinates of the first contour point and the position offsets of other contour points other than the first contour point relative to the first contour point can be determined as contour point data corresponding to the first image frame.

[0085] By recording the contour point data as the position coordinates of the first contour point and the position offsets of other contour points relative to the first contour point, the data volume of the contour point data is reduced, thereby reducing the computing pressure of the first device and improving data transmission efficiency.

[0086] Step 230: Send the first image frame and the contour point data corresponding to the first image frame to the first device.

[0087] The second device sends the first image frame and the contour point data corresponding to the first image frame to the server through the network, and then the server sends the first image frame and the contour point data corresponding to the first image frame to the first device.

[0088] The technical solution provided in the embodiment of the present application detects the first image frame of the first video on the second device (such as the anchor terminal device or the server) to obtain the contour line of the first object contained in the first image frame and the N contour points on the contour line, so that after the contour point data corresponding to the first image frame and the N contour points are sent to the first device (such as the audience terminal device), the first device can determine the contour area of ​​the first object in the first image frame according to the first image frame and the corresponding contour point data. Compared with the related art in which the first device detects the first image frame of the first video, the technical solution provided in the present application transfers the steps of detecting the first object and extracting contour points to the second device, thereby avoiding a large computing pressure on the first device, reducing the computing resources required by the first device, and improving the operating efficiency of the first device.

[0089] Please refer to Figure 4 , which shows a flow chart of a method for acquiring contour point data provided by an embodiment of the present application. The execution subject of each step of the method may be a second device. The method may include at least one of the following steps 410 to 430:

[0090] Step 410: extract M contour points on the contour line, where M is a positive integer greater than N.

[0091] Optionally, the M contour points may refer to all pixel points on the contour line, or may refer to some pixel points on the contour line, may refer to some pixel points evenly distributed on the contour line, or may refer to some pixel points randomly extracted from the contour line.

[0092] In the embodiment of the present application, the M contour points refer to all pixel points on the contour line.

[0093] Step 410 includes at least one sub-step among steps 411 to 413 (not shown in the figure).

[0094] Step 411, obtaining a contour image corresponding to the first image frame, wherein the contour image is an image in which the inside and outside of the contour line in the first image frame are represented by different colors, wherein the inside of the contour line is represented by a first color, and the outside of the contour line is represented by a second color.

[0095] The contour image is used to characterize the contour area of ​​the first object in the first image frame, that is, the image composed of pixels represented by the first color is the contour area of ​​the first object, and the image composed of pixels represented by the second color is the background area in the first image frame, where the background area refers to other image areas in the first image frame except the contour area of ​​the first object.

[0096] refer to Figure 5 The contour image 500 corresponding to the first image frame shown is an image in which the inside and outside of the contour line in the first image frame are represented by black and white colors. The pixel points inside the contour line are represented by white 502, that is, the first color is white, and the pixel points outside the contour line are represented by black 501, that is, the second color is black.

[0097] In some embodiments, an image detection method may be used to obtain a contour image corresponding to the first image frame. For example, Faster R-CNN (Faster Region-Convolutional Neural Network, a region-based convolutional neural network algorithm) may be used to obtain a contour image corresponding to the first image frame.

[0098] First, the first image frame can be input into CNN (Convolutional Neural Network) to extract the feature image corresponding to the first image frame, and the feature image is used to characterize the features of each pixel in the first image frame. Secondly, based on the feature image, the image area that may contain the first object in the first image frame is extracted. Then, each extracted image area is classified into two categories to further determine whether each image area contains the first object, and the candidate contour line of the first object is extracted from the image area containing the first object. Finally, the candidate contour line of the first object is further adjusted to obtain a more accurate target contour line, and the final target contour line is used as the contour line of the first object in the first image frame.

[0099] According to the contour line of the first object obtained above, the contour area within the contour line and the background area outside the contour line in the first image frame are separated, and the pixel points of the contour area within the contour line are represented by the first color, and the pixel points of the background area outside the contour line are represented by the second color, so as to obtain the contour image corresponding to the first image frame, such as Figure 5 For example, taking Mask R-CNN as an example, a Mask branch can be added on the basis of Faster R-CNN, and the regional image output by Faster R-CNN and the corresponding feature image are used as the input of the convolution layer to obtain the mask image of the first image frame, that is, the contour image corresponding to the first image frame.

[0100] Optionally, before separating the contour area and the background area in the first image frame, in order to simplify and speed up the image processing process, the first image frame can be converted into a grayscale image. One conversion method is to obtain a grayscale value by weighted averaging RGB according to the sensitivity of the human eye to different color channels. For example, the grayscale value can be expressed as Gray = 0.299*R+0.587*G+0.114*B. Whether to process the image frame as a grayscale image can be selected according to actual business needs. For example, if some businesses only need to extract the outline of the object, the image can be processed as a grayscale image.

[0101] Step 412: for each row of pixels in the contour image, two pixels at the junction of the first color and the second color are determined as two contour points.

[0102] For each row of pixels in the contour image, optionally, each row of pixels can be traversed from left to right, and the first pixel point that changes from the second color to the first color is determined as the left contour point on the contour line, and the first pixel point that changes from the first color to the second color is determined as the right contour point on the contour line. The left contour point and the right contour point are the two pixel points at the junction of the first color and the second color.

[0103] For each row of pixel points in the contour image, optionally, each row of pixel points in the contour image can be traversed from right to left, and the first pixel point that changes from the second color to the first color is determined as the right contour point on the contour line, and the first pixel point that changes from the first color to the second color is determined as the left contour point on the contour line. The right contour point and the left contour point are the two pixel points at the junction of the first color and the second color.

[0104] Optionally, for each column of pixels in the contour image, two pixels at the junction of the first color and the second color may be determined as two contour points. Each column of pixels may be traversed from top to bottom, and the first pixel that changes from the second color to the first color may be determined as an upper contour point on the contour line, and the first pixel that changes from the first color to the second color may be determined as a lower contour point on the contour line, and the upper contour point and the lower contour point are the two pixels at the junction of the first color and the second color. Each column of pixels may also be traversed from bottom to top, and the first pixel that changes from the second color to the first color may be determined as a lower contour point on the contour line, and the first pixel that changes from the first color to the second color may be determined as an upper contour point on the contour line, and the lower contour point and the upper contour point are the two pixels at the junction of the first color and the second color.

[0105] Step 413, obtaining M contour points according to the two contour points respectively determined from each row of pixel points in the contour image.

[0106] By traversing each row of pixels in the contour image and obtaining the left contour points and right contour points corresponding to each row of pixels, the M contour points in the contour image can be obtained. The contour point data corresponding to the M contour points can be expressed as:

[0107] {[Line1,(x left_1 ,y left_1 ),(x right_1 ,y right_1 )],

[0108] …[Line i ,(x left_i ,y left_i ),(x right_i ,y right_i )],

[0109] …[Line Q ,(x left_Q ,y left_Q ),(x right_Q ,y right_Q )]}

[0110] Among them, [Line1, (x left_1 ,y left_1 ),(x right_1 ,y right_1 )] represents the left and right contour points corresponding to the first row of pixels in the contour image, (x left_1 ,y left_1 ) represents the contour point data of the left contour point corresponding to the first row of pixels, (x right_1 ,y right_1) represents the contour point data of the right contour point corresponding to the first row of pixels, [Line i ,(x left_i ,y left_i ),(x right_i ,y right_i )] represents the left contour point and the right contour point corresponding to the pixel points in the i-th row in the contour image. According to the left contour point and the right contour point corresponding to the pixel points in the Q-th row, M contour points can be obtained.

[0111] By traversing each row of pixel points in the contour image, two pixel points at the junction of the first color and the second color are determined as two contour points, all contour points in the contour point image are extracted more accurately, and contour point data of M contour points are obtained, thereby better indicating the position of the first object in the first image frame.

[0112] Step 420, calculating the sampling step length according to the number M of contour points on the contour line and the number N of contour points to be extracted; wherein N is a preset fixed value.

[0113] M is the number of all pixel points on the contour line in the first image frame, which is a randomly changing value. Therefore, by setting N to a preset fixed value, after obtaining M contour points on the contour line, the sampling step can be calculated according to the number of contour points N that need to be extracted. The sampling step can be expressed as step = M / N.

[0114] Furthermore, since N is a pre-set fixed value, the data volume of the contour point data corresponding to the N contour points always remains unchanged, thereby reducing the computing pressure of the second device and avoiding the situation where the data volume of the contour point data is too large due to N being too large, and avoiding the situation where the N contour points cannot accurately indicate the position of the first object in the first image frame due to N being too small.

[0115] Step 430 , sampling N contour points from the M contour points according to the sampling step length, to obtain contour point data corresponding to the first image frame.

[0116] According to the sampling step length, the M contour points are sampled to obtain N contour points, thereby obtaining contour point data corresponding to the N contour points, that is, the contour point data corresponding to the first image frame.

[0117] In order to reduce the amount of data transmitted by encoding, the number N of contour points may be set according to actual business scenarios. For example, N may be set to 64 or 128.

[0118] The schematic diagram of the contour points in the first image frame can be referred to Figure 6As shown, part of the contour points 601 after sampling are shown. The contour points 601 are pixel points at the connection position of the first color and the second color in the contour image. Several contour points 601 can roughly represent the position of the first object in the first image frame.

[0119] The technical solution provided in the embodiment of the present application obtains N contour points evenly distributed on the contour line by extracting M contour points from the contour image and sampling the M contour points according to the sampling step size, so that the position of the first object in the first image frame can be indicated by the N contour points, thereby avoiding the transmission burden on the second device due to the excessive amount of contour point data of the M contour points, and also improving the processing efficiency of the second device.

[0120] Next, the process of acquiring the first image frame is introduced.

[0121] In some embodiments, an image frame is extracted from the first video every T image frames as the first image frame, where T is a positive integer.

[0122] In a live broadcast scene, the changes of the first object in the picture are usually relatively stable, and generally no large motion scenes will appear. In addition, for the embodiment of the present application, the contour line is applied to the first device, and the bullet screen information of the first image frame is displayed outside the contour line of the first object. A certain error between the contour line and the actual object contour is allowed. Based on this, the above-mentioned image detection, image segmentation, and contour line extraction process can be performed once every T frames according to actual business needs to reduce the consumption of computing resources.

[0123] Correspondingly, the contour point data corresponding to the remaining image frames between two adjacent first image frames can adopt the contour point data corresponding to the previous first image frame. Therefore, the position of the first object in the first image frame in the remaining image frames between two adjacent first image frames can refer to the position of the first object in the first image frame indicated by N contour points in the previous first image frame.

[0124] In some embodiments, T is a value that changes dynamically based on changes in content of the first video.

[0125] Exemplarily, if the motion changes of the first object in the first video are relatively frequent, or the amplitude of the motion changes is large, it is necessary to speed up the image detection frequency, that is, reduce the value of T, to avoid the contour point data in the currently extracted first image frame being significantly different from the contour point data in the image frame where image detection has not been performed, resulting in the contour line in the currently extracted first image frame being unable to represent the contour line of the first object in the image frame where image detection has not been performed.

[0126] Exemplarily, if the motion change of the first object in the first video is small, or the amplitude of the motion change is small, it is necessary to reduce the image detection frequency, that is, increase the value of T, so as to reduce the consumption of computing resources.

[0127] In some embodiments, T changes dynamically within a certain range, for example, T may change dynamically within a range from 5 to 20.

[0128] In some embodiments, for two adjacent first image frames extracted, at least one pair of contour points located at the same position in the sequence are obtained from the first contour point sequence and the second contour point sequence; the distance value between the two contour points included in each pair of contour points is calculated, and the distance value is used to characterize the difference in the positions of the two contour points in the first image frame; and T is adjusted according to the distance value.

[0129] Among them, the first contour point sequence is a contour point sequence corresponding to one of two adjacent first image frames, the second contour point sequence is a contour point sequence corresponding to the other of two adjacent first image frames, and the contour point sequence is a sequence obtained by arranging N contour points extracted from the first image frames in a set order.

[0130] For the two adjacent first image frames extracted, the distance value between the two contour points contained in each pair of contour points in the two first image frames before and after is calculated. The distance value can be expressed as: Wherein, (x1, y1) represents the position coordinates of the contour point included in one of the two adjacent first image frames, and (x2, y2) represents the position coordinates of the contour point included in the other of the two adjacent first image frames.

[0131] According to the size of the distance value of each pair of contour points in two adjacent first image frames, the position difference of the contour points in the two adjacent first image frames can be determined, so that the value of T can be adjusted according to the distance value of each pair of contour points, so that the distance value of each pair of contour points in the two adjacent first image frames can be maintained within a dynamic range.

[0132] In some embodiments, if there is a number of distance values ​​greater than a first threshold that satisfies a first number, the value of T is reduced.

[0133] For the distance values ​​of each pair of contour points in two adjacent first image frames, exemplarily, if the number of distance values ​​greater than the first threshold is greater than or equal to the first number, it means that the motion change of the first object in the first video is large, and the position difference of the first object in adjacent first image frames is large. Therefore, it is necessary to increase the frequency of image detection to avoid the contour point data corresponding to the previous first image frame of two adjacent first image frames cannot be used to represent the position of the first object in the image frame in the image frame where image detection is not performed between them. In this case, the value of T needs to be reduced.

[0134] It should be noted that the first threshold and the first number here can be set according to actual business needs, and this application does not limit this. For example, the first number can be N / 4, and the first threshold can be 1 / 100 of the width of the first image frame. If the distance value exceeding N / 4 is greater than 1 / 100 of the width of the first image frame, the value of T is reduced.

[0135] In some embodiments, if there is a number of distance values ​​less than a second threshold that satisfies a second number, the value of T is increased.

[0136] For the distance values ​​of each pair of contour points in two adjacent first image frames, exemplarily, if the number of distance values ​​less than the second threshold is greater than or equal to the second number, it means that the motion change of the first object in the first video is small, and the position difference of the first object in adjacent first image frames is small, so the frequency of image detection needs to be reduced, the position of the first object in two adjacent first image frames does not change significantly, and the contour point data corresponding to the previous first image frame in the two adjacent first image frames can be used to represent the position of the first object in more image frames, then the value of T needs to be increased.

[0137] It should be noted that the second threshold and the second number here can be set according to actual business needs, and this application does not limit this. Optionally, the second threshold and the first threshold can be the same or different, and the second number and the first number can be the same or different. For example, the second number can be set to N / 8, and the second threshold can be 1 / 100 of the width of the first image frame. If the distance value that does not exceed N / 8 is greater than 1 / 100 of the width of the first image frame, the value of T is increased.

[0138] Optionally, the distance values ​​of each pair of contour points in adjacent first image frames may be calculated continuously, and if the number of distance values ​​that are calculated multiple times in a row and that are less than the second threshold is greater than or equal to the second number, the value of T is increased. For example, if the distance value that is calculated three times in a row and that does not exceed N / 8 is greater than 1 / 100 of the width of the first image frame, the value of T is increased.

[0139] In some embodiments, if the number of distance values ​​greater than the first threshold does not satisfy the first number, and the number of distance values ​​less than the second threshold does not satisfy the second number, the value of T remains unchanged.

[0140] You can refer to Figure 7 As shown, the distance value between the two contour points included in each pair of contour points in two adjacent first image frames is calculated. If the number of distance values ​​greater than the first threshold satisfies the first number, the value of T is reduced. Otherwise, it is determined whether the number of distance values ​​less than the second threshold satisfies the second number. If so, the value of T is increased. Otherwise, the value of T is kept unchanged. That is, if the number of distance values ​​greater than the first threshold does not satisfy the first number, and the number of distance values ​​less than the second threshold does not satisfy the second number, the value of T is kept unchanged.

[0141] Next, the implementation process of step 230 is introduced. Step 230 includes at least one sub-step of steps 231 to 233 (not shown in the figure).

[0142] Step 231: encode the first image frame to obtain first code stream data.

[0143] The first code stream data refers to the data unit obtained after encoding the first image frame. When encoding the first image frame, the first image frame is encoded into multiple macroblocks, and the macroblocks are further divided into multiple NALUs (Network Abstraction Layer Units). After encoding multiple NALUs, the first code stream data can be obtained. The code stream format of NALU can be referred to as shown in Table 1 below.

[0144] Table 1

[0145]

[0146] Among them, the data corresponding to the start code (Start Code) is 0x00000001, and the data corresponding to the NALU header is the data type corresponding to the NALU. For example, if the NALU is SEI (Supplemental Enhancement Information), the data corresponding to the NALU header is 0x06; if the NALU is a data unit encoded by an I frame, the data corresponding to the NALU header is 0x25 or 0x65, and the specific data is related to the I frame type; if the NALU is a data unit encoded by a P frame, the data corresponding to the NALU header is 0x21 or 0x61, and the specific data is related to the P frame type. The data corresponding to the NALU body will be filled in according to the encoded data format, and different types of NALUs correspond to encoded data formats.

[0147] Step 232: encapsulate the contour point data corresponding to the first image frame into supplemental enhancement information SEI, and encode the SEI to obtain second code stream data.

[0148] Supplementary enhancement information SEI is a mechanism for adding additional information during the transmission of video bitstreams. SEI is part of the H.264 standard and can be encoded and decoded through extended syntax to encapsulate auxiliary information into a specific type of NALU. SEI can encapsulate other information related to the video content, such as video copyright information, audio sampling rate, video subtitles, lens switching during video content generation, screen rotation, etc. In step 232, the contour point data corresponding to the first image frame is encapsulated into SEI, and SEI is encoded to obtain the second bitstream data.

[0149] Optionally, the second code stream data may be inserted before or after the first code stream data corresponding to any I frame or P frame. After the second code stream data is inserted into the first code stream data, the contour point data corresponding to the second code stream data may be used only to represent the contour point data corresponding to the first image frame, or may be used to represent the contour point data corresponding to all image frames between the first image frame and the next first image frame for image detection.

[0150] The NALU body corresponding to SEI will be filled in according to the data format encoded by SEI. The data format corresponding to SEI can be referred to as shown in Table 2 below.

[0151] Table 2

[0152]

[0153] Among them, NAL type refers to the Network Abstract Layer (NAL) type, the data corresponding to NAL type is 0x06, and the data length is 1 byte; the data corresponding to Playload type is 0x05, and the data length is 1 byte; UUID (Universally Unique Identifier) ​​and custom data length will be calculated from custom data, and the type format of custom data is specified by the business party; the data corresponding to the end alignment code of SEI is 0x80, and the data length is 1 byte.

[0154] The transmission format of SEI's custom data may be referred to as shown in Table 3 below.

[0155] Table 3

[0156]

[0157] Among them, the data corresponding to business_type (business type) will be defined according to the business type of SEI encapsulation information. For example, the business type of the contour point data corresponding to the image frame is defined as 0x01, and the business type of the speech translation of the image frame is defined as 0x02. point_count indicates the number of contour points corresponding to the first image frame, and the corresponding data can be set according to the number of the first contour points. start_x and start_y respectively indicate the position coordinates of the first contour point among the N contour points, and the corresponding data can be set according to the position coordinates of the first contour point. x_diff_uint and y_diff_uint respectively indicate the number of bits of the position offset of the other contour points other than the first contour point relative to the first contour point in the clockwise or counterclockwise direction on the contour line, and the corresponding data can be set according to their respective position offsets, wherein x_diff and y_diff respectively indicate the position offset of the other contour points other than the first contour point among the N contour points relative to the first contour point. x_diff_data and y_diff_data respectively indicate the bit stream data of x_diff and y_diff arranged according to x_diff_uint and y_diff_uint.

[0158] Because the coordinate values ​​of contour points are usually represented by int data, which are usually represented by 4 bytes. The contour area of ​​the first object is limited in the image frame. Therefore, when calculating the position offset of each contour point relative to the first contour point to represent other contour points except the first contour point, the position offset will be concentrated in a smaller range than the size range of the entire first image frame. The reduced range is reflected in a smaller value, so that the smaller value can be represented by fewer bits, thereby achieving the purpose of compressing data. For example, originally, 2 int data (i.e. 8 bytes, 64 bits) were required to represent a contour point, but after calculating the position offset, only a few bits are needed to express a contour point.

[0159] Step 233: Send a video code stream of the first video to the first device, where the video code stream includes first code stream data and second code stream data.

[0160] The first code stream data and the second code stream data are sent to the first device. The second code stream data can be inserted before or after the first code stream data, which is not limited in the present application.

[0161] By encapsulating the contour point data corresponding to the first image frame into SEI and using position offsets to represent the position coordinates of each contour point, the amount of data corresponding to the contour point data is reduced as much as possible, avoiding increasing the video encoding bit rate due to excessive data volume, thereby affecting the transmission of the video bit stream and affecting the viewing effect of the video on the first device.

[0162] Please refer to Figure 8 , which shows a flow chart of an image processing method provided by another embodiment of the present application. The execution subject of each step of the method may be a first device. The method may include at least one of the following steps 810 to 830:

[0163] Step 810, receiving a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame; wherein the contour point data is used to indicate the positions of N contour points on a contour line of a first object contained in the first image frame in the first image frame, and N is a positive integer.

[0164] In some embodiments, a video code stream of a first video sent by a second device is received, the video code stream including first code stream data and second code stream data; the first code stream data is decoded to obtain a first image frame; the second code stream data is decoded to obtain supplemental enhancement information SEI, and contour point data corresponding to the first image frame is obtained from SEI.

[0165] SEI decoding is the reverse process of SEI encoding. When receiving NALU code stream in real time, after discovering the Start Code, the type of NALU is determined based on the subsequent type bytecode. After obtaining the NALU of SEI, the custom data can be parsed according to the decoding rules.

[0166] The relevant introduction in step 810 can be referred to the above embodiment, which will not be repeated here.

[0167] Step 820: Determine a contour line of a first object contained in the first image frame according to the contour point data.

[0168] According to the positions of the N contour points in the first image frame, a contour line based on the N contour points may be determined, and the contour line may be used to indicate a contour area of ​​the first object in the first image frame.

[0169] In some embodiments, N contour points are interpolated according to the contour point data to obtain P contour points, where P is a positive integer greater than N; and a contour line of a first object included in the first image frame is determined according to the P contour points.

[0170] Commonly used interpolation algorithms include but are not limited to linear interpolation, spline interpolation, Bezier curve interpolation and other algorithms. The corresponding algorithm can be selected according to the actual scene requirements.

[0171] Taking linear interpolation as an example, we can insert a coordinate point for every two contour points. For example, if (x left_1 ,y left_1 ) and (x right_1 ,y right_1 ) is linearly interpolated, the interpolated coordinates are (x left_1.5 ,y left_1.5 ), where the horizontal and vertical coordinates can be expressed as:

[0172] x left_1.5 =x left_1 +(x left_2 -x left_1 ) / 2

[0173] y left_1.5 =y left_1 +[(y left_2 -y left_1 ) / (x left_2 -x left_1 )]*(x left_1.5 -x left_1 )

[0174] Taking N as 64 as an example, after interpolation using the above linear interpolation method, the value of P is 128.

[0175] Optionally, several coordinate points may be inserted for every two contour points, so that P contour points far more than N contour points may be obtained.

[0176] By interpolating the N contour points, more contour points are obtained, so that the contour line of the first object can be determined according to the P contour points, so that the obtained contour line is closer to the contour line of the first object in the first image frame, thereby improving the accuracy of the contour line.

[0177] In some embodiments, an image frame is extracted from every T image frames in the first video as the first image frame, where T is a positive integer. If the image frame to be displayed is the first image frame, step 820 is executed.

[0178] The image frame to be displayed refers to an image frame received by the first device and not yet displayed on the client of the first device.

[0179] If the image frame to be displayed is the first image frame, that is, the image frame to be displayed is an image frame on which the second device performs image detection and obtains its corresponding contour point data, the first device will interpolate the N contour points according to the contour point data corresponding to the image frame to be displayed, and determine the contour line of the first object contained in the image frame to be displayed based on the P contour points after interpolation.

[0180] In some embodiments, if the image frame to be displayed is not the first image frame, the contour line of the first object contained in the most recent first image frame before the image frame to be displayed is determined as the contour line of the first object contained in the image frame to be displayed.

[0181] If the image frame to be displayed is not the first image frame, then the image frame to be displayed is the image frame between two adjacent first image frames. The first device may determine the contour line of the first object contained in the most recent first image frame before the image frame to be displayed as the contour line of the first object contained in the image frame to be displayed.

[0182] Exemplarily, if the first device sequentially receives frame A, frame B, frame C, and frame D, wherein frame A and frame D are first image frames, then if the image frame to be displayed is frame A or frame D, the contour line of the first object contained in frame A or frame D is determined as the contour line of the first object contained in the image frame to be displayed. If the image frame to be displayed is frame B or frame C, the contour line of the first object contained in frame A is determined as the contour line of the first object contained in the image frame to be displayed.

[0183] Step 830: display the first image frame, and display the associated information of the first image frame obtained based on the contour line of the first object.

[0184] The associated information of the first image frame includes, but is not limited to, the barrage information of the first image frame, the gift information of the first image frame, the advertising information of the first image frame, the voting information of the first image frame, etc.

[0185] In some embodiments, the associated information of the first image frame includes bullet-chat information of the first image frame. Then step 830 includes: displaying the first image frame, and displaying the bullet-chat information of the first image frame outside the contour line of the first object.

[0186] The bullet chat information is generally drawn on a canvas, so the contour area corresponding to the contour line of the bullet chat information of the first image frame can be cropped, and the bullet chat information can be drawn to the outer area of ​​the contour line of the first object on the canvas while displaying the first image frame.

[0187] Fig. 9 A schematic diagram of the display effect after the barrage is cropped is shown, where a first image frame 901 is displayed on the client of the first device, the first image frame including an outline 903 of a first object, and while the first image frame is displayed, barrage information 902 of the first image frame is displayed outside the outline 903 of the first object.

[0188] Fig.10The specific implementation process diagram of the image processing method is shown. The second device performs image detection on the first image frame of the first video to obtain a contour image corresponding to the first image frame. Optionally, the contour image can be converted into a grayscale image first, and then the contour area and background area in the first image frame are segmented to obtain the contour line of the first object contained in the first image frame. After encoding the first image frame and the contour point data corresponding to the first image frame, the video code stream is sent to the first device. The first device decodes the video code stream and interpolates the contour point data therein to obtain a contour line that better fits the contour of the first object, thereby cropping the contour area corresponding to the contour line and displaying the bullet information outside the contour line of the first object.

[0189] The technical solution provided by the embodiment of the present application allows the first device to directly determine the contour line of the first object contained in the first image frame based on the contour point data by receiving the first image frame processed by the second device and the contour point data corresponding to the first image frame, so that the associated information of the first image frame obtained based on the contour line of the first object can be displayed. The steps of detecting the first object and extracting the contour points are transferred to the second device, avoiding the first device performing image processing and causing a large computing pressure on the first device, thereby reducing the computing resources required by the first device and improving the operating efficiency of the first device.

[0190] Fig.11 The flowchart of the image processing method after the second device and the first device are combined is shown. The second device performs image detection on the first image frame, extracts the contour image corresponding to the first image frame, and then extracts N contour points in the contour image according to the contour image, encapsulates the contour point data corresponding to the N contour points into SEI, encodes them into second code stream information, and sends them to the first device together with the first code stream information corresponding to the first image frame. The first device decodes the video code stream to obtain contour point data, performs interpolation processing on the N contour points, and determines the contour line of the first object according to the interpolated contour points. Finally, the bullet screen information of the first image frame can be displayed outside the contour line of the first object while displaying the first image frame.

[0191] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0192] Please refer to Fig.12 , which shows a block diagram of an image processing device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned image processing method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the second device introduced above, or it can be set in the second device. Fig.12As shown, the device 1200 may include: a detection module 1210 , an extraction module 1220 and a sending module 1230 .

[0193] The detection module 1210 is configured to detect a first image frame of a first video to obtain a contour line of a first object contained in the first image frame.

[0194] The extraction module 1220 is used to extract N contour points on the contour line to obtain contour point data corresponding to the first image frame, where the contour point data is used to indicate positions of the N contour points in the first image frame, and N is a positive integer.

[0195] The sending module 1230 is used to send the first image frame and the contour point data corresponding to the first image frame to the first device.

[0196] In some embodiments, the extraction module 1220 includes:

[0197] The extraction unit is used to extract M contour points on the contour line, where M is a positive integer greater than N.

[0198] A sampling unit is used to sample the N contour points from the M contour points according to a sampling step size to obtain contour point data corresponding to the first image frame.

[0199] In some embodiments, the extraction unit is used to:

[0200] Acquire a contour image corresponding to the first image frame, wherein the contour image is an image in which the inside and outside of the contour line in the first image frame are represented by different colors, wherein the inside of the contour line is represented by a first color, and the outside of the contour line is represented by a second color;

[0201] For each row of pixels in the contour image, two pixels at the positions where the first color and the second color meet are determined as two contour points;

[0202] The M contour points are obtained according to the two contour points respectively determined from each row of pixel points in the contour image.

[0203] In some embodiments, the sampling unit is further used for:

[0204] The sampling step length is calculated according to the number M of contour points on the contour line and the number N of contour points to be extracted; wherein N is a preset fixed value.

[0205] In some embodiments, the apparatus further comprises an image frame extraction module, wherein the image frame extraction module is configured to:

[0206] From the first video, an image frame is extracted every T image frames as the first image frame, where T is a value that changes dynamically based on changes in content of the first video, and T is a positive integer.

[0207] In some embodiments, the device further comprises an adjustment module, wherein the adjustment module is configured to:

[0208] For two adjacent first image frames extracted, at least one pair of contour points located at the same position in the sequence is obtained from the first contour point sequence and the second contour point sequence; wherein the first contour point sequence is a contour point sequence corresponding to one of the two adjacent first image frames, the second contour point sequence is a contour point sequence corresponding to the other of the two adjacent first image frames, and the contour point sequence is a sequence obtained by arranging the N contour points extracted from the first image frames in a set order;

[0209] Calculating a distance value between two contour points included in each pair of contour points, wherein the distance value is used to characterize a difference in positions of the two contour points in the first image frame;

[0210] T is adjusted according to the distance value.

[0211] In some embodiments, the adjustment module is used to:

[0212] If the number of distance values ​​greater than the first threshold satisfies the first number, then reduce the value of T;

[0213] If the number of distance values ​​less than the second threshold satisfies the second number, increase the value of T;

[0214] If the number of distance values ​​greater than the first threshold does not satisfy the first number, and the number of distance values ​​less than the second threshold does not satisfy the second number, the value of T remains unchanged.

[0215] In some embodiments, the apparatus further comprises an encoding module, wherein the encoding module is configured to:

[0216] Encoding the first image frame to obtain first code stream data;

[0217] Encapsulating the contour point data corresponding to the first image frame into supplemental enhancement information SEI, and encoding the SEI to obtain second bitstream data;

[0218] Sending a video code stream of the first video to the first device, where the video code stream includes the first code stream data and the second code stream data.

[0219] In some embodiments, the contour point data corresponding to the first image frame includes: the position coordinates of a first contour point among the N contour points, and the position offsets of other contour points among the N contour points except the first contour point relative to the first contour point.

[0220] In some embodiments, the first video is a live video, the method is executed by a second device, the second device is a host-end device corresponding to the live video, and the first device is a viewer-end device corresponding to the live video.

[0221] The technical solution provided in the embodiment of the present application detects the first image frame of the first video on the second device (such as the anchor terminal device or the server) to obtain the contour line of the first object contained in the first image frame and the N contour points on the contour line, so that after the contour point data corresponding to the first image frame and the N contour points are sent to the first device (such as the audience terminal device), the first device can determine the contour area of ​​the first object in the first image frame according to the first image frame and the corresponding contour point data. Compared with the related art in which the first device detects the first image frame of the first video, the technical solution provided in the present application transfers the steps of detecting the first object and extracting contour points to the second device, thereby avoiding a large computing pressure on the first device, reducing the computing resources required by the first device, and improving the operating efficiency of the first device.

[0222] Please refer to Fig.13 , which shows a block diagram of an image processing device provided by another embodiment of the present application. The device has the function of implementing the above-mentioned image processing method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the first device introduced above, or it can be set in the first device. Fig.13 As shown, the device 1300 may include: a receiving module 1310 , a determining module 1320 and a display module 1330 .

[0223] The receiving module 1310 is used to receive a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame; wherein the contour point data is used to indicate the positions of N contour points on the contour line of a first object contained in the first image frame in the first image frame, and N is a positive integer.

[0224] The determination module 1320 is configured to determine a contour line of the first object contained in the first image frame according to the contour point data.

[0225] The display module 1330 is configured to display the first image frame and display associated information of the first image frame obtained based on the contour line of the first object.

[0226] In some embodiments, the determining module 1320 is used to:

[0227] According to the contour point data, interpolation processing is performed on the N contour points to obtain P contour points, where P is a positive integer greater than N;

[0228] A contour line of the first object contained in the first image frame is determined according to the P contour points.

[0229] In some embodiments, the receiving module 1310 is used to:

[0230] receiving a video code stream of the first video sent by the second device, wherein the video code stream includes first code stream data and second code stream data;

[0231] Decoding the first code stream data to obtain the first image frame;

[0232] The second code stream data is decoded to obtain supplemental enhancement information SEI, and contour point data corresponding to the first image frame is obtained from the SEI.

[0233] In some embodiments, an image frame is extracted from the first video every T image frames as the first image frame, where T is a positive integer; and the display module 1330 is used to:

[0234] If the image frame to be displayed is the first image frame, performing the step of determining the contour line of the first object contained in the first image frame according to the contour point data;

[0235] If the image frame to be displayed is not the first image frame, the contour line of the first object contained in the first image frame most recently before the image frame to be displayed is determined as the contour line of the first object contained in the image frame to be displayed.

[0236] In some embodiments, the associated information of the first image frame includes bullet screen information of the first image frame; the display module 1330 is used to:

[0237] The first image frame is displayed, and barrage information of the first image frame is displayed outside the outline of the first object.

[0238] The technical solution provided by the embodiment of the present application allows the first device to directly determine the contour line of the first object contained in the first image frame based on the contour point data by receiving the first image frame processed by the second device and the contour point data corresponding to the first image frame, so that the associated information of the first image frame obtained based on the contour line of the first object can be displayed. The steps of detecting the first object and extracting the contour points are transferred to the second device, avoiding the first device performing image processing and causing a large computing pressure on the first device, thereby reducing the computing resources required by the first device and improving the operating efficiency of the first device.

[0239] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0240] Please refer to Fig.14 , which shows a block diagram of a computer device 1400 provided in one embodiment of the present application. The computer device 1400 may be any electronic device with data calculation, processing and storage functions. The computer device 1400 may be used to implement the image processing method provided in the above embodiment.

[0241] Typically, the computer device 1400 includes a processor 1401 and a memory 1402 .

[0242] Processor 1401 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. Processor 1401 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, processor 1401 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, processor 1401 may also include an AI processor for processing computing operations related to machine learning.

[0243] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1402 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned image processing method.

[0244] Those skilled in the art will understand that Fig.14 The structure shown in the figure does not constitute a limitation on the computer device 1400, and the computer device 1400 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0245] In an illustrative embodiment, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is executed by a processor of a computer device, the image processing method is implemented. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0246] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned image processing method.

[0247] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting relevant data of users and during the process of collecting relevant data of users. The prompt interface, pop-up window or voice prompt information is used to prompt the user that relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are processed strictly in accordance with the requirements of relevant national laws and regulations, and the informed consent or separate consent of the subject of personal information is obtained with the consent and authorization of the user. The subsequent data use and processing behavior is carried out within the scope of authorization of laws and regulations and the subject of personal information, and the collection, use and processing of relevant user data shall comply with the relevant laws, regulations and standards of relevant countries and regions.

[0248] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0249] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Detecting a first image frame of a first video to obtain a contour line of a first object contained in the first image frame; Extracting N contour points on the contour line to obtain contour point data corresponding to the first image frame, wherein the contour point data is used to indicate positions of the N contour points in the first image frame, where N is a positive integer; The first image frame and contour point data corresponding to the first image frame are sent to a first device.

2. The method according to claim 1, characterized in that The extracting N contour points on the contour line to obtain contour point data corresponding to the first image frame includes: Extracting M contour points on the contour line, where M is a positive integer greater than N; The N contour points are sampled from the M contour points according to a sampling step size to obtain contour point data corresponding to the first image frame.

3. The method according to claim 2, characterized in that The extracting M contour points on the contour line comprises: Acquire a contour image corresponding to the first image frame, wherein the contour image is an image in which the inside and outside of the contour line in the first image frame are represented by different colors, wherein the inside of the contour line is represented by a first color, and the outside of the contour line is represented by a second color; For each row of pixels in the contour image, two pixels at the positions where the first color and the second color meet are determined as two contour points; The M contour points are obtained according to the two contour points respectively determined from each row of pixel points in the contour image.

4. The method according to claim 2, characterized in that: The method further comprises: The sampling step length is calculated according to the number M of contour points on the contour line and the number N of contour points to be extracted; wherein N is a preset fixed value.

5. The method according to claim 1, characterized in that: The method further comprises: From the first video, an image frame is extracted every T image frames as the first image frame, where T is a value that changes dynamically based on changes in content of the first video, and T is a positive integer.

6. The method according to claim 5, characterized in that The method further comprises: For two adjacent first image frames extracted, at least one pair of contour points located at the same position in the sequence is obtained from the first contour point sequence and the second contour point sequence; wherein the first contour point sequence is a contour point sequence corresponding to one of the two adjacent first image frames, the second contour point sequence is a contour point sequence corresponding to the other of the two adjacent first image frames, and the contour point sequence is a sequence obtained by arranging the N contour points extracted from the first image frames in a set order; Calculating a distance value between two contour points included in each pair of contour points, wherein the distance value is used to characterize a difference in positions of the two contour points in the first image frame; T is adjusted according to the distance value.

7. The method according to claim 6, characterized in that The adjusting T according to the distance value includes: If the number of distance values ​​greater than the first threshold satisfies the first number, then reduce the value of T; If the number of distance values ​​less than the second threshold satisfies the second number, increase the value of T; If the number of distance values ​​greater than the first threshold does not satisfy the first number, and the number of distance values ​​less than the second threshold does not satisfy the second number, the value of T remains unchanged.

8. The method according to any one of claims 1 to 7, characterized in that: The sending the first image frame and the contour point data corresponding to the first image frame to the first device includes: Encoding the first image frame to obtain first code stream data; Encapsulating the contour point data corresponding to the first image frame into supplemental enhancement information SEI, and encoding the SEI to obtain second bitstream data; Sending a video code stream of the first video to the first device, where the video code stream includes the first code stream data and the second code stream data.

9. The method according to any one of claims 1 to 7, characterized in that: The contour point data corresponding to the first image frame includes: the position coordinates of a first contour point among the N contour points, and the position offsets of other contour points among the N contour points except the first contour point relative to the first contour point.

10. The method according to any one of claims 1 to 7, characterized in that: The first video is a live video, and the method is executed by a second device, the second device is a host-end device corresponding to the live video, and the first device is a viewer-end device corresponding to the live video.

11. An image processing method, characterized in that: The method comprises: Receive a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame; wherein the contour point data is used to indicate positions of N contour points on a contour line of a first object contained in the first image frame in the first image frame, where N is a positive integer; determining a contour line of the first object contained in the first image frame according to the contour point data; The first image frame is displayed, and associated information of the first image frame obtained based on the contour line of the first object is displayed.

12. The method according to claim 11, characterized in that Determining the contour line of the first object contained in the first image frame according to the contour point data includes: According to the contour point data, interpolation processing is performed on the N contour points to obtain P contour points, where P is a positive integer greater than N; A contour line of the first object contained in the first image frame is determined according to the P contour points.

13. The method according to claim 11, characterized in that The receiving a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame, includes: receiving a video code stream of the first video sent by the second device, wherein the video code stream includes first code stream data and second code stream data; Decoding the first code stream data to obtain the first image frame; The second code stream data is decoded to obtain supplemental enhancement information SEI, and contour point data corresponding to the first image frame is obtained from the SEI.

14. The method according to claim 11, characterized in that An image frame is extracted from the first video every T image frames as the first image frame, where T is a positive integer; and the method further includes: If the image frame to be displayed is the first image frame, performing the step of determining the contour line of the first object contained in the first image frame according to the contour point data; If the image frame to be displayed is not the first image frame, the contour line of the first object contained in the first image frame most recently before the image frame to be displayed is determined as the contour line of the first object contained in the image frame to be displayed.

15. The method according to any one of claims 11 to 14, characterized in that The associated information of the first image frame includes bullet comment information of the first image frame; The displaying of the first image frame and the displaying of the associated information of the first image frame obtained based on the contour line of the first object include: The first image frame is displayed, and barrage information of the first image frame is displayed outside the outline of the first object.

16. An image processing device, characterized in that: The device comprises: A detection module, configured to detect a first image frame of a first video to obtain a contour line of a first object contained in the first image frame; An extraction module, used to extract N contour points on the contour line to obtain contour point data corresponding to the first image frame, wherein the contour point data is used to indicate positions of the N contour points in the first image frame, where N is a positive integer; A sending module is used to send the first image frame and the contour point data corresponding to the first image frame to the first device.

17. An image processing device, characterized in that: The device comprises: A receiving module, configured to receive a first image frame of a first video sent by a second device, and contour point data corresponding to the first image frame; wherein the contour point data is used to indicate positions of N contour points on a contour line of a first object contained in the first image frame in the first image frame, where N is a positive integer; a determination module, configured to determine a contour line of the first object contained in the first image frame according to the contour point data; The display module is configured to display the first image frame and display associated information of the first image frame obtained based on the contour line of the first object.

18. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the image processing method according to any one of claims 1 to 10, or to implement the image processing method according to any one of claims 11 to 15.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 10, or to implement the image processing method according to any one of claims 11 to 15.

20. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is loaded and executed by a processor to implement the image processing method according to any one of claims 1 to 10, or to implement the image processing method according to any one of claims 11 to 15.

Images