Method, device, electronic device and storage medium for detecting content delivered by delivery terminal
By obtaining the delivery video of the delivery terminal and performing image frame intercept and matching the content, and using the frame identification number to determine the delivery time, the problem of wide distribution range and high manual audit cost of the delivery terminal is solved, and automatic monitoring of the delivery of multimedia information is realized and Comprehensive monitoring.
Patent Information
- Application Number
- CN202011129946.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-10-21
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-10-21
AI Technical Summary
The delivery terminal has a wide distribution range, making it difficult to comprehensively monitor the delivery of multimedia information, and the cost of manual auditing is high.
By obtaining the delivery video of the delivery terminal, intercepting the image frame and matching the content, the frame identification number is used to determine the delivery time of the multimedia information, and automatic monitoring is achieved.
There is no need for a dedicated person to monitor in real time, which reduces the cost of manual auditing and realizes comprehensive monitoring of the delivery of multimedia information.
Smart Images

Figure CN114386997B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of multimedia information technology, and in particular to a method, device, electronic device and computer-readable storage medium for detecting content delivered by a delivery terminal. Background Art
[0002] In the field of multimedia information, using delivery terminals (such as advertising machines) to deliver multimedia information (such as advertising images or advertising videos) to demanders (such as merchants) is an emerging and important information delivery channel.
[0003] The delivery terminals are flexible and can be freely distributed in different locations according to application requirements, such as elevators, subways, bus stations, etc., to improve the dissemination of information. However, at the same time, the delivery terminals have a wide distribution range and the delivery content is not easy to monitor and audit.
[0004] In the traditional method of monitoring and auditing the delivery content, people go to the distribution locations of the delivery terminals manually to audit the delivery content of the delivery terminals to determine whether the delivery terminals have delivered the multimedia information of the demanders, the number of times the multimedia information of the demanders has been delivered, the delivery duration, and other delivery conditions.
[0005] However, due to the wide distribution of delivery terminals, it is difficult to comprehensively monitor the delivery of multimedia information to demanders, and the cost of manual auditing is high. Summary of the invention
[0006] The present application provides a method, device, electronic device and computer-readable storage medium for detecting content delivered by a delivery terminal, aiming to solve the problem that due to the wide distribution range of delivery terminals, it is difficult to comprehensively monitor the delivery status of the multimedia information to be delivered and the manual audit cost is high.
[0007] In a first aspect, the present application provides a method for detecting content delivered by a delivery terminal, the method comprising:
[0008] Get the delivery video of the delivery terminal;
[0009] According to a preset interception frequency, the image frames of the delivery video are intercepted to obtain N frames of delivery images of the delivery terminal, wherein each frame of the N frames of delivery images is provided with a frame identification number, and the frame identification number is used to indicate the delivery time sequence of the N frames of delivery images;
[0010] Performing content matching on each of the N frames of delivered images to obtain a target image whose delivered content matches the multimedia information to be delivered;
[0011] The delivery duration of the multimedia information is determined according to the frame identification number, the target image and the interception frequency.
[0012] In a second aspect, the present application provides a delivery terminal delivery content detection device, the delivery terminal delivery content detection device comprising:
[0013] An acquisition unit, used for acquiring the delivery video of the delivery terminal;
[0014] An interception unit, used to intercept the image frames of the delivery video according to a preset interception frequency to obtain N frames of delivery images of the delivery terminal, wherein each frame of the N frames of delivery images has a frame identification number, and the frame identification number is used to indicate the delivery time sequence of the N frames of delivery images;
[0015] A matching unit, configured to perform content matching on each of the N frames of delivered images to obtain a target image whose delivered content matches the multimedia information to be delivered;
[0016] A determination unit is used to determine the delivery duration of the multimedia information according to the frame identification number, the target image and the interception frequency.
[0017] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the processor executes the steps of any one of the methods for detecting content delivered by a delivery terminal provided in the present application.
[0018] In a fourth aspect, the present application further provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the method for detecting content delivered by a delivery terminal.
[0019] This application obtains the delivery video of the delivery terminal and performs image frame capture to obtain N frames of delivery images of the delivery terminal; and performs content matching on each frame of the N frames of delivery images to obtain a target image whose delivery content matches the multimedia information to be delivered, so as to detect whether the multimedia information to be delivered is delivered on the delivery terminal; and the delivery duration of the multimedia information can be determined according to the frame identification number, the target image and the capture frequency. It can be seen that, on the one hand, since there is no need for dedicated personnel to monitor in real time, the problem of wide distribution range of delivery terminals, difficulty in comprehensively monitoring the delivery of multimedia information and high cost of manual auditing is solved. On the other hand, it can automatically monitor whether the delivery terminal has delivered the multimedia information to be delivered, as well as the delivery duration of the multimedia information, so the delivery of multimedia information can be monitored more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0021] Figure 1 This is a schematic diagram of a scenario of a delivery terminal delivery content detection system provided in an embodiment of the present application;
[0022] Figure 2 This is a flow chart of a method for detecting content delivered by a delivery terminal provided in an embodiment of the present application;
[0023] Figure 3 It is a schematic diagram of an image of a certain frame in the delivery video provided in an embodiment of the present application;
[0024] Figure 4 is a schematic flow chart of an embodiment of step S40 provided in an embodiment of the present application;
[0025] Figure 5 is a schematic flow chart of an embodiment of step S43 provided in an embodiment of the present application;
[0026] Figure 6 This is a schematic diagram of a scenario of a delivery terminal provided in an embodiment of the present application;
[0027] Figure 7 It is a schematic diagram of the structure of an embodiment of a delivery terminal delivery content detection device provided in an embodiment of the present application;
[0028] Figure 8 It is a schematic diagram of the structure of an embodiment of an electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0030] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0031] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.
[0032] The embodiment of the present application provides a method, device, electronic device and computer readable storage medium for detecting content delivered by a delivery terminal. The delivery terminal delivery content detection device can be integrated in an electronic device, which can be a server or a terminal.
[0033] First of all, before introducing the embodiments of the present application, the relevant content about the application background of the embodiments of the present application is introduced.
[0034] In the multimedia information field, it is increasingly common to use delivery terminals (such as advertising machines) to deliver multimedia information, such as product advertisements by merchants and public service advertisements by government departments.
[0035] Many companies will entrust advertisers to place advertisements for their own products in places such as elevator advertising machines and road sign advertising spots. Many of them are in the form of videos. Due to the wide distribution of the placement locations and the various forms of placement, the companies have to set up their own manual teams to review whether the advertisements have been placed in designated locations and the effectiveness of the placements. This requires huge manpower and material resources and has a low input-output ratio.
[0036] Based on the above-mentioned defects in the existing related technologies, the embodiments of the present application provide a method for detecting content delivered by a delivery terminal, which at least overcomes the defects in the existing related technologies to a certain extent.
[0037] The executor of the delivery terminal delivery content detection method of the embodiment of the present application may be the delivery terminal delivery content detection device provided in the embodiment of the present application, or different types of electronic devices such as a server device, a physical host or a user equipment (UE) that integrates the delivery terminal delivery content detection device, wherein the delivery terminal delivery content detection device may be implemented in hardware or software, and the UE may specifically be a terminal device such as a smart phone, a tablet computer, a laptop computer, a PDA, a desktop computer or a personal digital assistant (PDA).
[0038] The electronic device can work in a single operation mode or in a device cluster mode. By applying the delivery terminal delivery content detection method provided in the embodiment of the present application, it can automatically monitor whether the delivery terminal has delivered the multimedia information to be delivered and the delivery time of the multimedia information. No dedicated person is required for real-time monitoring, which solves the problem that the delivery terminals are widely distributed, it is difficult to comprehensively monitor the delivery of multimedia information, and the cost of manual audit is high.
[0039] See also Figure 1 , Figure 1 It is a scene diagram of the delivery terminal delivery content detection system provided in the embodiment of the present application. Among them, the delivery terminal delivery content detection system may include an electronic device 100, and the delivery terminal delivery content detection device is integrated in the electronic device 100. For example, the electronic device can obtain the delivery video of the delivery terminal; according to the preset interception frequency, the image frame of the delivery video is intercepted to obtain N frames of delivery images of the delivery terminal; for each frame of the N frames of delivery images, the content is matched respectively to obtain the target image whose delivery content is consistent with the multimedia information to be delivered; according to the frame identification number, the target image and the interception frequency, the delivery duration of the multimedia information is determined.
[0040] In addition, if Figure 1 As shown, the delivery terminal delivery content detection system may further include a memory 200 for storing data, such as image data and video data.
[0041] It should be noted that Figure 1 The scenario diagram of the delivery terminal delivery content detection system shown is only an example. The delivery terminal delivery content detection system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the delivery terminal delivery content detection system and the emergence of new business scenarios, the technical solution provided by the embodiment of the present invention is also applicable to similar technical problems.
[0042] Next, the delivery terminal delivery content detection method provided in the embodiment of the present application is introduced. In the embodiment of the present application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.
[0043] Reference Figure 2 , Figure 2 It is a flow chart of a method for detecting the delivery content of a delivery terminal provided in an embodiment of the present application. It should be noted that although the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that here. The method for detecting the delivery content of a delivery terminal includes steps S10 to S40, wherein:
[0044] S10: Obtain the delivery video of the delivery terminal.
[0045] Among them, the delivery terminal can be in various forms, such as advertising machines, mobile TVs, cash registers, vending machines, etc. The delivery terminals can be distributed in subways, buses, bus stops, elevators, hotel lobbies, shopping malls, etc. Of course, the above delivery terminal forms and distribution locations are only examples. In the embodiments of the present application, the delivery terminals can have more forms and distribution locations, which are not limited here.
[0046] The delivery video refers to a video reflecting the delivery terminal delivering content, which may include the original video of the delivered content or a video shot based on the delivered content. Figure 3 As shown, Figure 3 It is a schematic diagram of an image of a certain frame in the delivered video provided in an embodiment of the present application.
[0047] In some embodiments, the content actually delivered in the delivery terminal can be directly retrieved as the delivery video. For example, the electronic device executing the embodiment of the present application is directly connected to the advertising machine in the elevator, and the electronic device can directly record or retrieve any video played by the advertising machine as the delivery video.
[0048] In some embodiments, each delivery terminal is covered by the field of view of the corresponding surveillance camera, and the surveillance camera can record the delivery content of the delivery terminal by collecting video. After collecting the video of the delivery terminal, the surveillance camera can upload the video of the delivery terminal to the electronic device. The electronic device can directly use the video of the delivery terminal collected by the surveillance camera as the delivery video.
[0049] For example, there are usually advertising machines and cameras in elevators. The cameras in the elevator can monitor the content of the advertising machine by collecting videos. At this time, the video of the advertising machine collected by the camera in the elevator can be used as the video for delivery.
[0050] In some embodiments, in order to ensure monitoring strength, save manpower, and reduce monitoring costs, passers-by (such as pedestrians, express delivery personnel, and takeaway delivery personnel) can directly shoot a video facing the delivery terminal to record the delivery content of the delivery terminal. After the passer-by shoots the video of the delivery terminal, the video of the delivery terminal can be uploaded to the electronic device. The electronic device can directly use the video of the delivery terminal shot by the passer-by as the delivery video. For example, in order to save manpower, the express delivery personnel can use the mobile phone to shoot a 3 to 5 minute video of the passing advertising machine when delivering the express as the delivery video.
[0051] S20. According to a preset interception frequency, image frames are intercepted for the delivery video to obtain N frames of delivery images of the delivery terminal.
[0052] Wherein, N is a positive integer. Each frame of the N frames of delivered images has a frame identification number, and the frame identification number is used to indicate the delivery time sequence of the N frames of delivered images.
[0053] For example, the frame rate of the original delivery video is 25 frames per second. With a capture frequency of one frame every 30 frames, a total of 1800 frames of delivery video are captured to obtain 60 frames of delivery images of the delivery terminal. The frame identification numbers of the 60 frames of delivery images are recorded in sequence according to the delivery time: 1, 2, 3, 4, ..., N. The 60 frames of images can represent the set Q = [1, 2, 3, 4, ..., N], where N = 60.
[0054] For another example, a 3-minute video is captured at a frame rate of 5 seconds, and 36 frames of images are captured for the video. The frame identification numbers of the 36 frames are recorded in chronological order as: 1, 2, 3, 4, ..., N. The 36 frames can represent the set Q = [1, 2, 3, 4, ..., N], where N = 36.
[0055] The above-mentioned interception frequency is only an example and can be set according to actual conditions without limitation.
[0056] S30 , performing content matching on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered.
[0057] Each frame image refers to the j-th frame image (also referred to as “j-th frame image” in this article) among N frames of projection images, wherein the value range of j is 1 to N.
[0058] The multimedia information to be delivered (if not otherwise specified, referred to herein as "multimedia information" means the multimedia information to be delivered) refers to the multimedia information that the demander needs to disseminate. The demander may be a business, a government department, etc. For example, the multimedia information may be a product advertisement of a business, a public service advertisement of a government department, etc. It is understandable that the multimedia information and demander here are only examples, and there may be more specific forms of expression, which are not limited here.
[0059] The target image refers to an image in the N frames of delivered images whose delivered content matches the multimedia information to be delivered. For example, if the multimedia information to be delivered is a milk powder advertisement of merchant A, if the delivered content corresponding to the j-th frame image is also a milk powder advertisement of merchant A, then the j-th frame image is the target image. In the embodiment of the present application, the purpose of determining the target image is to find the image frame whose delivered content is the multimedia information to be delivered, so as to determine whether the multimedia information is delivered.
[0060] In some embodiments, the text information in the jth frame of the N frames of delivery images is extracted and compared with the text information of the multimedia information. When the text information in the jth frame is the same as the text information of the multimedia information, the jth frame is used as the target image whose delivery content is consistent with the multimedia information.
[0061] In some embodiments, pattern information in the jth frame of N frames of delivery images is extracted and compared with the pattern information of the multimedia information. When the pattern information in the jth frame is the same as the pattern information of the multimedia information, the jth frame is used as the target image whose delivery content is consistent with the multimedia information.
[0062] In some embodiments, the text information and pattern information in the jth frame of the N frames of delivered images are extracted and compared with the text information and pattern information of the multimedia information. When the text information in the jth frame is the same as the text information of the multimedia information, and the pattern information is the same as the pattern information of the multimedia information, the jth frame is used as the target image whose delivery content is consistent with the multimedia information.
[0063] S40: Determine the delivery duration of the multimedia information according to the frame identification number, the target image and the interception frequency.
[0064] The delivery duration refers to the delivery duration of multimedia information on the delivery terminal during a delivery video period of the delivery terminal.
[0065] In some embodiments, the total number of image frames of the target image is directly counted, and then the delivery duration of the multimedia information is calculated based on the total number of image frames, the capture frequency, and the frame rate of the delivered video.
[0066] In some embodiments, the delivery terminal is deemed to have effectively delivered the multimedia information to be delivered only when the continuous delivery exceeds a certain duration. First, the continuous delivery segments of the multimedia information are found according to the frame identification number and the target image, and then the total number of images of the continuous delivery segments is counted. Finally, the delivery duration of the multimedia information is calculated according to the total number of images of the continuous delivery segments, the interception frequency, and the frame rate of the delivery video.
[0067] The related concepts of continuous delivery of segments are introduced in detail later in the text and will not be elaborated here.
[0068] From the above content, it can be seen that by acquiring the delivery video of the delivery terminal and performing image frame capture, N frames of delivery images of the delivery terminal are obtained; and each frame of the N frames of delivery images is matched with the content to obtain the target image whose delivery content is consistent with the multimedia information to be delivered, so as to detect whether the multimedia information to be delivered is delivered on the delivery terminal; by using the frame identification number, the target image and the capture frequency, the delivery duration of the multimedia information can be determined. It can be seen that, on the one hand, since there is no need for real-time monitoring by a dedicated person, the problem of wide distribution range of delivery terminals, difficulty in comprehensively monitoring the delivery of multimedia information and high cost of manual auditing is solved. On the other hand, it can automatically monitor whether the delivery terminal has delivered the multimedia information to be delivered, as well as the delivery duration of the multimedia information, so the delivery of multimedia information can be monitored more comprehensively.
[0069] In some scenarios, the multimedia information to be delivered contains specific text and images. By detecting whether each frame of the N delivered images contains the corresponding text and images, it can be determined whether the delivery terminal is delivering the multimedia information. In this case, step S30 may specifically include the following steps a1 to a3, wherein:
[0070] a1. Obtaining reference text information of the multimedia information.
[0071] The reference text information refers to at least one of specific text and specific patterns in the multimedia information to be delivered.
[0072] For example, the multimedia information to be delivered is a milk powder advertisement of merchant A, and the base text information contained in the milk powder advertisement is: "such and such milk powder", "national sales leader", "double certification" and other words, as well as a cow image.
[0073] a2. Perform text detection processing on each frame of image.
[0074] In some embodiments of the present application, an existing text recognition algorithm may be used to perform text detection on the j-th frame image of N frames of delivery images to obtain a set W of text information of the j-th frame image. j=[w1, w2, ..., wn], where n is the amount of text information in the j-th frame image, and n is a positive integer.
[0075] In some embodiments of the present application, a text recognition network for text detection is also provided. Specifically, the j-th frame image of the N-frame delivery image is input into the text recognition network provided in the embodiment of the present application, so that the text recognition network performs text detection on the j-th frame image, and obtains a set W of text information of the j-th frame image. j =[w1,w2,...,wn].
[0076] The above text recognition network can be trained through the following steps:
[0077] 1. Build a preliminary text recognition network.
[0078] For example, the preliminary text recognition network may include a text detection module and a text recognition module, and the open source EAST network may be used as the text detection module and the open source DRCNN network may be used as the text recognition module. The text detection module is used to detect the detection frame where the text is located in the image, and the text recognition module is used to recognize the text based on the detection frame where the text is located to obtain the text information of the image.
[0079] 2. Get the training dataset.
[0080] The training data set includes multiple sample images, a portion of the sample images may be images containing reference text information, and a portion of the sample images may be images containing other scene texts.
[0081] 3. Use the training data set to train the preliminary text recognition network until the preliminary text recognition network converges to obtain the trained text recognition network.
[0082] Among them, the trained text detection module can fully learn the relationship between the location of the text and the image features, so that the detection box of the location of the text in the image can be accurately detected. The trained text recognition module can fully learn the relationship between the image features and the text at the location of the text, so that the text information in the image can be accurately detected.
[0083] Among them, the training process of the text recognition network is similar to the training process of the existing network model. For the training process that is not described in detail, you can refer to the training method of the existing network model and will not go into details here.
[0084] a3. When target text information of each frame image is detected, each frame image is used as the target image.
[0085] The target text information refers to the text information in each frame of image that is the same as the reference text information.
[0086] Specifically, when it is detected that the j-th frame image has target text information, the j-th frame image is used as the target image.
[0087] For easier understanding, the example of step a1 above is continued.
[0088] In a specific example, the set of text information W of the j-th frame image j If any text information wj is the same as any text in the reference text information, it can be determined that the target text information exists in the j-th frame image.
[0089] For example, the set of text information of the j-th frame image W j =[w1, w2], where w1 = "such and such milk powder", w2 = "leading in national sales". Then it can be determined that the target text information exists in the j-th frame image, and the j-th frame image is the target image.
[0090] In a specific example, the set of text information W of the j-th frame image j When all the text information in the reference text information is included in the j-th frame image, it is determined that the target text information exists.
[0091] For example, the set of text information of the j-th frame image W j =[w1, w2, w3], where w1 = "such and such milk powder", w2 = "national sales leader", and w3 = "double certification", it can be determined that the target text information exists in the j-th frame image, and the j-th frame image is the target image.
[0092] For example, the set of text information of the j-th frame image W j =[w1, w2], where w1 = "such and such milk powder" and w2 = "leading in national sales", it can be determined that there is no target text information in the j-th frame image, and the j-th frame image is not the target image.
[0093] For example, the set of text information of the j-th frame image W j =[w1, w2, w3], where w1 = "such and such milk powder", w2 = "national sales leader", and w3 = "infant milk powder", it can be determined that there is no target text information in the j-th frame image, and the j-th frame image is not the target image.
[0094] From the above content, it can be seen that by detecting whether there is target text information in each frame image that is the same as the actual text and pattern of the multimedia information (i.e., the reference text information), it is determined whether each frame image contains multimedia information, and then the target image containing multimedia information can be quickly and accurately found.
[0095] In some embodiments, an image in the multimedia information to be delivered can be intercepted as a reference image, and the similarity between each frame of the N frames of delivered images and the reference image can be directly compared. If the similarity is high, it can be determined that the delivery terminal is delivering the multimedia information. At this time, correspondingly, step S30 can specifically include the following steps b1 to b2, wherein:
[0096] b1. Obtaining the similarity between each frame image and a preset reference image.
[0097] The reference image matches the multimedia information. The reference image is an image pre-captured from the multimedia information.
[0098] For example, the feature vector of the j-th frame image can be extracted through the trained open source mobilenetv2 network. Then, according to the feature vector of the j-th frame image, cosine similarity is used for matching, so that the similarity between the j-th frame image and the preset reference image can be obtained. Among them, matching by cosine similarity is an existing similarity matching algorithm, which will not be repeated here.
[0099] b2. When the similarity is greater than a preset similarity threshold, each frame of the image is used as the target image.
[0100] For example, the preset similarity threshold is 80%, if the similarity of the j-th frame image is 90%, the j-th frame image can be used as the target image. If the similarity of the j-th frame image is 60%, the j-th frame image can be determined as not being the target image.
[0101] The above preset similarity threshold is only an example and can be set according to actual needs without limitation.
[0102] From the above content, it can be seen that by comparing each frame of image with the similarity of the reference image pre-captured from the multimedia information, it is determined whether each frame of image has been delivered with multimedia information, and then the target image with multimedia information delivered can be found quickly and accurately.
[0103] Due to blurring and shaking of some videos, the text in the jth frame image may not be clear. In this case, the above steps a1 to a3 will misjudge the jth frame image as not being the target image (i.e., the content being delivered will be misjudged as not being the multimedia information to be delivered), thereby missing a part of the target image, resulting in a low recall rate for delivery content detection.
[0104] In view of this situation, in some embodiments, the target image is determined in the manner of steps a1 to a3, and the above steps b1 to b2 can also be used as auxiliary judgment. Specifically, if the j-th frame image is determined as not being the target image in steps a1 to a3, the above steps b1 to b2 can be further judged to determine whether the j-th frame image is the target image.
[0105] Furthermore, if steps a1 to a3 determine that the j-th frame image is not the target image and steps b1 to b2 determine that the j-th frame image is the target image, the confidence level of determining that the j-th frame image is the target image in steps b1 to b2 is obtained. If the confidence level is greater than a preset confidence threshold, the j-th frame image is ultimately determined to be the target image.
[0106] Please refer to Figure 4 , Figure 4 is a flow chart of an embodiment of step S40 provided in the embodiments of the present application. In some embodiments of the present application, step S40 may specifically include S41 to S43, wherein:
[0107] S41, obtaining a target frame identification number of the target image.
[0108] For ease of understanding, the example in the above step S20 is continued.
[0109] For example, 60 frames of images can represent a set Q = [1, 2, 3, 4, ..., N], where N = 60. Among them, the images with frame identification numbers: 6-14, 38-47 are target images, and the target frame identification numbers are: 6-14, 38-47.
[0110] For another example, 36 frames of images can be represented by a set Q = [1, 2, 3, 4, ..., N], where N = 36. The images with frame identification numbers 6 to 14 are target images, and the target frame identification numbers are 6 to 14.
[0111] S42: Determine the start frame and the end frame of the continuous delivery of the multimedia information according to the target frame identification number.
[0112] The start frame refers to the start frame of the continuous delivery segment of the multimedia information continuously delivered on the delivery terminal. The end frame refers to the end frame of the continuous delivery segment of the multimedia information continuously delivered on the delivery terminal. The number of continuous delivery segments is ≥1, and correspondingly, the number of start frames is ≥1 and the number of end frames is ≥1.
[0113] For easier understanding, the description will continue with the example in the above step SS41.
[0114] Specifically, among N frames of images, if at least two consecutive frames of images are target images, it can be determined that the multimedia information is continuously delivered, and the at least two consecutive frames of images can constitute a continuous delivery segment.
[0115] Among them, image continuity refers to the image frames with the closest delivery time among the N frames of delivery images. Whether the image is continuous can be determined by the frame identification number. For example, the frame identification numbers of N=60 frames of delivery images are recorded in sequence according to the delivery time: 1, 2, 3, 4, ..., N, and the 60 frames of images can represent the set Q=[1, 2, 3, 4, ..., N], then 1, 2 and 3 are 3 consecutive frames of images, and 5, 6, 7, 8 and 9 are 5 consecutive frames of images. If 1, 2, 3, 5, 6, 7, 8, and 9 are all target images, then 1, 2, and 3 image frames can constitute a continuous delivery segment, and 5, 6, 7, 8, and 9 can constitute a continuous delivery segment.
[0116] Among them, in N frames of images, the number of the continuous delivery segments may be x, where x is a positive integer greater than or equal to 1. Each of the continuous delivery segments may be recorded as an "i-th delivery segment", where i is in the range of 1 to x.
[0117] In a specific example, the delivery video contains only one continuous delivery segment of multimedia information, that is, in this delivery video, the delivery terminal delivers the multimedia information to be delivered once. For example, in a 3-minute delivery video shot by a courier deliveryman at the delivery terminal, a milk powder advertisement of merchant A is delivered once. Among them, one delivery corresponds to one start frame and one end frame.
[0118] For example, 36 frames of images can represent a set Q = [1, 2, 3, 4, ..., N], where N = 36. The target frame identification numbers are: 6-14, then it can be determined that the continuous delivery segment is 1, the start frame of the continuous delivery segment is 6, and the end frame is 14.
[0119] In some embodiments, the delivery video includes multiple continuous delivery segments of multimedia information, that is, in the delivery video, the delivery terminal delivers the multimedia information to be delivered multiple times. For example, in a 3-minute delivery video shot by a courier deliveryman at the delivery terminal, the milk powder advertisement of merchant A is delivered twice. Each delivery has a corresponding start frame and end frame. That is, there are multiple start frames and end frames for the continuous delivery of multimedia information.
[0120] For example, 60 frames of images can represent a set Q = [1, 2, 3, 4, ..., N], where N = 60. The target frame identification numbers are: 6-14, 38-47. Then it can be determined that the continuous delivery segments are 2, where the start frame of the first delivery segment is 6 and the end frame is 14, and the start frame of the second delivery segment is 38 and the end frame is 47.
[0121] S43: Determine the delivery duration of the multimedia information according to the start frame, the end frame and the interception frequency.
[0122] For example, if the frame rate of the original delivery video is 25 frames per second, and a frame is captured every 30 frames, a total of 1800 frames of delivery video is captured, and N = 60 frames of delivery images are obtained for the delivery terminal. The 60 frames of images can represent the set Q = [1, 2, 3, 4, ..., N].
[0123] In N=60 frames of images, there may be two continuous delivery segments. The first delivery segment has a start frame of 1 and an end frame of 14, and the second delivery segment has a start frame of 38 and an end frame of 47.
[0124] Then, the duration of the first delivery segment is: t1 = 14*30*(1 / 25) = 16.8 seconds.
[0125] The duration of the second delivery segment is: t2 = (47-38+1)*30*(1 / 25) = 12 seconds.
[0126] Finally, it can be determined that the delivery duration of the multimedia information is: t1+t2=28.8 seconds.
[0127] From the above content, it can be seen that by considering the continuous delivery segments as effective delivery, finding out the x continuous delivery segments delivered continuously on the delivery terminal, and then counting the delivery duration of the multimedia information based on the image frames contained in the x continuous delivery segments, the actual delivery situation of the delivery terminal can be monitored more effectively and comprehensively.
[0128] Since the target image mentioned in step S30 may be misjudged, there is a possibility that the image frame of the multimedia content may be misjudged as not being the target image. If this happens, the content that originally belonged to a continuous delivery segment will be split into two segments.
[0129] Taking into account that in the process of delivering multimedia information, the possibility of inserting other information in a short period of time is small, therefore, if the number of image frames between two consecutive delivery segments is less than the preset frame number threshold (such as only 1 frame between segment 1 and segment 2), it can be determined that the two consecutive delivery segments belong to the same continuous segment, thereby avoiding to a certain extent the problem of reduced accuracy of delivery content detection due to image frames of multimedia content being misjudged as not being the target image.
[0130] Please refer to Figure 5 , Figure 5 It is a flowchart of an embodiment of step S43 provided in the embodiments of the present application.
[0131] To this end, in some embodiments of the present application, in a delivery video of the delivery terminal, the number of continuous delivery segments is x, the number of the start frames and the number of the end frames are both x, where x is a positive integer greater than 1, and step S43 may specifically include S431 to S435, where:
[0132] S431: Use the image frames between the i-th start frame and the i-th end frame as the continuous delivery segments of the multimedia information.
[0133] The value of i ranges from 1 to x, and x represents the number of consecutively delivered segments before updating (ie, before merging the segments).
[0134] For example, the frame rate of the original delivery video is 25 frames per second. With a capture frequency of one frame every 30 frames, a video with a total of 1800 frames is captured to obtain N = 60 frames of delivery images of the delivery terminal. The frame identification numbers of the 60 frames of delivery images are recorded in sequence according to the delivery time: 1, 2, 3, 4, ..., N. The 60 frames of images can represent the set Q = [1, 2, 3, 4, ..., N].
[0135] Among them, image frames 3 to 8, 10 to 14, and 38 to 50 are target images, and the number of continuous delivery segments before updating (i.e., before segment merging) is 3. The start frame of the first continuous delivery segment (i.e., the first delivery segment) is 3, and the end frame is 8. The start frame of the second continuous delivery segment (i.e., the second delivery segment) is 10, and the end frame is 14. The start frame of the third continuous delivery segment (i.e., the third delivery segment) is 38, and the end frame is 50.
[0136] S432, obtaining the first image frame number between the (i-1)th ending frame and the i-th starting frame.
[0137] The first number of image frames refers to the number of continuous image frames between the end frame of the (i-1)th continuous delivery segment and the start frame of the i-th continuous delivery segment.
[0138] For ease of understanding, the above step S431 is continued. For example, if the number of image frames between the end frame (8) of the first continuous delivery segment and the start frame (10) of the second continuous delivery segment is 1, then the first image frame number is 1. If the number of image frames between the end frame (14) of the second continuous delivery segment and the start frame (38) of the third continuous delivery segment is 23, then the first image frame number is 23.
[0139] S433: When the first image frame number is less than a preset frame number threshold, update the continuous delivery segment according to the image frames between the (i-1)th start frame and the i-th end frame.
[0140] Specifically, when the first image frame number is less than the preset frame number threshold, the image frames between the start frame of the (i-1)th continuous delivery segment and the end frame of the i-th continuous delivery segment are merged into a new continuous delivery segment. When the first image frame number is greater than or equal to the preset frame number threshold, the (i-1)th continuous delivery segment and the i-th continuous delivery segment are not merged.
[0141] The above step S432 is continued. For example, the preset frame number threshold is 5.
[0142] Since the number of the first image frame is 1, which is less than the preset frame number threshold (5), the first continuous delivery segment and the second continuous delivery segment are merged into one segment; the specific method of merging is: the image frames between the start frame of the first continuous delivery segment and the end frame of the second continuous delivery segment are taken as a new continuous delivery segment to obtain a merged continuous delivery segment.
[0143] Since the number of frames of the first image is 23, which is greater than the preset frame number threshold (5), the second continuous delivery segment and the third continuous delivery segment are not merged.
[0144] The above preset frame number threshold is only an example and can be adjusted according to actual conditions and is not limited here.
[0145] S434: Obtain the updated second image frame number of the continuously delivered segment.
[0146] The second image frame number refers to the number of continuous image frames between the start frame of the i'th continuous delivery segment after update (i.e. after the segments are merged) and the end frame of the i'th continuous delivery segment after update. Wherein, i' ranges from 1 to x', and x' represents the number of continuous delivery segments after update.
[0147] The above step S433 example is continued. For example, after the merging process, the number of the updated continuous delivery segments is 2.
[0148] After the update (ie after the segments are merged), the start frame of the first continuously delivered segment is 3 and the end frame is 14, and the start frame of the second continuously delivered segment is 38 and the end frame is 50.
[0149] It can be determined that the number of continuous image frames between the start frame 3 and the end frame 14 of the updated first continuous delivery segment is 12, that is, the number of second image frames of the updated first continuous delivery segment is 12. The number of continuous image frames between the start frame 38 and the end frame 50 of the updated second continuous delivery segment is 23, that is, the number of second image frames of the updated second continuous delivery segment is 23.
[0150] S435: Determine the delivery duration of the multimedia information according to the second image frame number and the interception frequency.
[0151] Specifically, the frame rate of the delivered video is obtained. Then, the product of the second image frame number, the interception frequency and the inverse of the frame rate of the delivered video is used as the delivery duration of the multimedia information.
[0152] The above example of step S434 is continued.
[0153] For example, the number of updated continuous delivery segments is 2, the number of updated second image frames of the first continuous delivery segment is 12, and the number of updated second image frames of the second continuous delivery segment is 23.
[0154] It can be determined that the delivery duration of the multimedia information is: (12+23)*30*(1 / 25)=42 seconds.
[0155] From the above content, it can be seen that by merging two continuous delivery segments with a short time interval into one continuous delivery segment, it is avoided that the image frame of the delivered content is misjudged as not being the target image, which causes the actual continuous delivery segment to be split into two discontinuous segments, thereby improving the detection accuracy of the delivered content.
[0156] In order to understand the delivery status of the multimedia information to be delivered by the delivery terminal in more detail, in the embodiment of the present application, the duration of each continuous delivery segment can also be counted.
[0157] At this time, in some embodiments of the present application, step S435 may specifically include: determining the second duration of the continuous delivery segment according to the second image frame number and the interception frequency; and counting the delivery duration of the multimedia information according to the second duration.
[0158] The second duration refers to the duration of each continuous delivery segment.
[0159] For example, in the example of step S435, the duration of the first continuous delivery segment is: t1 = 12*30*(1 / 25) = 14.4 seconds.
[0160] The duration of the second continuous delivery segment is: t2 = 23*30*(1 / 25) = 27.6 seconds.
[0161] Finally, it can be determined that the delivery duration of the multimedia information is: t1+t2=42 seconds.
[0162] From the above content, it can be seen that by counting the second duration of each continuous delivery segment according to the second image frame number and the interception frequency, it can be reflected that when the delivery terminal delivers the multimedia information to be delivered, the duration of each continuous delivery segment can be reflected, and the delivery status of the multimedia information to be delivered by the delivery terminal can be more comprehensively detected.
[0163] On the basis of the above, in order to better reflect the delivery status of the multimedia information to be delivered by the delivery terminal, in some embodiments of the present application, the total duration of the delivered video and the delivery times of the media information in the delivered video can also be obtained. For example, the total duration of the delivered video is the first duration. At this time, the delivery terminal delivery content detection method also includes: counting the number of continuous delivery segments as the delivery times of the multimedia information within the first duration.
[0164] In some embodiments, the continuous delivery segments are not merged, and in this case, the number x of the continuous delivery segments can be directly used as the delivery number of the multimedia information in the first duration. Alternatively, the number x of the continuous delivery segments before the update (i.e., before the segments are merged) can be counted as the delivery number of the multimedia information in the first duration.
[0165] For example, in the example of step S43, there are two consecutive delivery segments, and the consecutive delivery segments are not merged. At this time, it can be determined that the delivery number of the multimedia information within the first duration is 2.
[0166] For another example, in step S431, the number of continuously delivered segments before updating (ie, before segment merging) is 3. At this time, it can be determined that the number of delivery times of the multimedia information within the first duration is 3.
[0167] In some embodiments, the number of the updated continuous delivery segments is counted as the number of times the multimedia information is delivered within the first duration.
[0168] For example, the number of continuously delivered segments before updating (ie before segment merging) is x=3, and the number of continuously delivered segments after updating (ie after segment merging) is x'=2, then the delivery times of the multimedia information in the first duration is determined to be 2.
[0169] It can be seen from the above content that by counting the number of continuous delivery segments as the delivery times of the multimedia information within the first duration, the delivery status of the multimedia information to be delivered by the delivery terminal can be more comprehensively detected.
[0170] Please refer to Figure 6 , Figure 6Schematic diagram of a scene of a delivery terminal provided in an embodiment of the present application. Some delivery terminals are divided into a static delivery area and a dynamic delivery area, wherein it can be understood that the dynamic delivery area is used for dynamically delivering a video of multimedia information, and the static delivery area is used for fixed delivery of a certain frame of multimedia information. Figure 6 As shown, Figure 6 The dotted box in the upper middle area represents the dynamic delivery area of the delivery terminal, and the dotted box in the lower middle area represents the static delivery area of the delivery terminal.
[0171] Only when both the static delivery area and the dynamic delivery area of the delivery terminal are delivering the multimedia information to be delivered, can it be determined that the delivery terminal is delivering the multimedia information. Therefore, it is necessary to detect the static delivery area and the dynamic delivery area respectively to determine whether the delivery terminal has delivered the multimedia information.
[0172] At this time, step S30 may specifically include the following steps c1 to c3, wherein:
[0173] c1. Extracting the static delivery area and the dynamic delivery area in each frame of the image.
[0174] For example, first, based on a training data set (comprising multiple images of a sample delivery terminal that is delivering information, wherein the sample delivery terminal is delivering information divided into dynamic area delivery and static area delivery), a preset area detection network is trained so that the trained area detection network learns the features of the static delivery area and the dynamic delivery area, thereby obtaining a trained area detection network (suitable for detecting static delivery areas and dynamic delivery areas based on images).
[0175] The preset region detection network may be an open source network model that can be used for detection tasks, such as an OverFeat network, a YOLOv network, etc. Specifically, an open source network (that can be used for detection tasks) with model parameters as default values may be used as the preset region detection network.
[0176] Then, the j-th frame image in the N-frame delivery image is input into the trained region detection network, so that the trained region detection network detects the j-th frame image and determines the static delivery area and the dynamic delivery area in the j-th frame image.
[0177] c2. Perform text detection processing on the static delivery area.
[0178] Specifically, "performing text detection processing on the static delivery area" in step c2 is similar to "performing text detection processing on each frame of the image" in the above step a2. For details, please refer to the description and examples of the above step a2, which will not be repeated here.
[0179] c3. When it is detected that the first text exists in the static delivery area, each frame of the image is used as the target image.
[0180] The first text is the same as the preset reference static text. The reference static text includes at least one of specific words and specific patterns in the multimedia information to be delivered.
[0181] Specifically, in step c3, "when the first text is detected in the static delivery area, each frame image is used as the target image" is similar to the above step a3, "when the target text information of each frame image is detected, each frame image is used as the target image". For details, please refer to the description and examples of the above step a3, which will not be repeated here.
[0182] For the target image determined in steps c1 to c3, the continuous delivery segment x of the delivery video may be determined according to the method in steps S41 to S43. Alternatively, the updated continuous delivery segment x' of the delivery video may be determined according to the method in steps S431 to S435.
[0183] The method further comprises the following steps d1 to d2, wherein:
[0184] d1. Perform text detection processing on the dynamic delivery areas of the image frames in the continuous delivery segment respectively to obtain the second text of the continuous delivery segment.
[0185] The second text is the same as the preset reference dynamic text. The reference dynamic text includes at least one of specific words and specific patterns in the multimedia information to be delivered.
[0186] Specifically, for x consecutive delivery segments (or, for x' consecutive delivery segments), text detection processing is performed on the dynamic delivery area of the j-th frame image in each consecutive delivery segment i to determine whether there is text that is the same as the preset benchmark dynamic text in the j-th frame image in each continuous delivery segment i.
[0187] And the text in the j-th frame image of the continuous delivery segment i that is the same as the preset reference dynamic text is used as the second text of the continuous delivery segment i, so as to obtain a plurality of second texts of the continuous delivery segment i.
[0188] For example, the benchmark dynamic texts include: "such and such milk powder", "national sales leader", "double certification", "milk powder from such and such region", and "certified by such and such standard".
[0189] Continuous delivery segment 1 includes 10 frames of images, 1, 2, 3, ..., 10, among which the 3rd, 5th, 6th and 7th frames contain texts identical to the baseline dynamic text (e.g., "such and such milk powder", "obtained such and such standard certification", "national sales leader", "such and such milk powder" respectively). Then, the texts identical to the baseline dynamic texts appearing in the 3rd, 5th, 6th and 7th frames can be used as the second text of continuous delivery segment 1 respectively.
[0190] Among them, "performing text detection processing on the dynamic delivery area of the j-th frame image in each continuous delivery segment i" is similar to "performing text detection processing on each frame image" in the above step a2. For details, please refer to the description and examples of the above step a2, which will not be repeated here.
[0191] d2. When the amount of the second texts is less than a preset amount threshold, the continuous delivery segments are discarded.
[0192] Specifically, in step d1, after determining a plurality of second texts of the continuous delivery segment i, the number of the second texts is counted. When the number of the second texts is less than a preset number threshold, the continuous delivery segment i is discarded to avoid determining the continuous delivery segment i only in the dynamic delivery area and the static delivery area as a segment of the multimedia information delivered by the delivery terminal.
[0193] When the number of the second texts is greater than or equal to the preset number threshold, it is proved that the delivery content of the dynamic delivery area is the multimedia information to be delivered, and the continuous delivery segment i is retained.
[0194] For example, in the example of step d1, the number of second texts of the continuous delivery segment 1 is 4. If the preset number threshold is 3, it is proved that the delivery content of the dynamic delivery area is the multimedia information to be delivered, and the continuous delivery segment 1 can be retained. If the preset number threshold is 5, it is proved that the delivery content of the dynamic delivery area is not the multimedia information to be delivered, and the continuous delivery segment 1 is discarded.
[0195] Furthermore, in order to improve the accuracy of the determination of the content delivered in the dynamic delivery area, the number of non-repeated second texts in the continuous delivery segment i can be counted. When the number of non-repeated second texts in the continuous delivery segment i is less than the preset number threshold, the continuous delivery segment is discarded. When the number of non-repeated second texts in the continuous delivery segment i is greater than or equal to the preset number threshold, the continuous delivery segment is retained.
[0196] For example, in the example of step d1 above, the number of non-repeated second texts in the continuous delivery segment 1 is 3. If the preset number threshold is 4, the continuous delivery segment 1 is discarded. If the preset number threshold is 3, the continuous delivery segment 1 can be retained.
[0197] The above preset quantity threshold is only an example and can be adjusted according to actual needs without limitation.
[0198] From the above content, it can be seen that for the continuous delivery segment i, by combining the delivery content of the dynamic delivery area and the static delivery area of the j-th frame image, it is determined whether the delivery terminal continuously delivers the segment i effectively, thereby avoiding the situation where the dynamic delivery area and the static delivery area do not deliver the multimedia information to be delivered at the same time.
[0199] In some cases, due to the different models of devices used to shoot the delivery video and the rotation of the device during shooting, in order to improve the recognition efficiency and accuracy of subsequent images, in some embodiments of the present application, it is necessary to correct the direction of the captured N frames of delivery images. That is, the delivery terminal delivery content detection method of the embodiment of the present application can further include: adjusting the direction of the N frames of delivery images to a unified direction.
[0200] In a specific example, a network can be pre-constructed for training to obtain a direction discrimination network that can be used to identify the direction of an image.
[0201] Then, the jth frame of the N-frame projection image is input into the direction discrimination network provided in the embodiment of the present application, so that the direction discrimination network identifies the direction of the jth frame of the image and obtains the image direction of the jth frame of the image. For example, the image direction of the jth frame of the image can be rotated by 0°, 90°, 180°, or 270° relative to the reference direction.
[0202] If the image direction of the j-th frame image is rotated relative to the reference direction, the image direction of the j-th frame image is adjusted so that the image direction of the j-th frame image is the same as the reference direction.
[0203] The above direction discrimination network can be trained through the following steps:
[0204] 1. Build a preliminary direction discrimination network.
[0205] For example, the open source Mobilenetv2 network can be used as a preliminary direction discrimination network for 4 categories, wherein the 4 categories include 4 categories rotated by 0°, 90°, 180°, and 270° relative to the reference direction.
[0206] 2. Get the training dataset.
[0207] Among them, the training data set contains multiple sample images, some of the sample images may be images rotated 0° relative to the reference direction, some of the sample images may be images rotated 90° relative to the reference direction, some of the sample images may be images rotated 180° relative to the reference direction, and some of the sample images may be images rotated 270° relative to the reference direction.
[0208] 3. Use the training data set to train the preliminary direction discrimination network until the preliminary direction discrimination network converges to obtain the trained direction discrimination network.
[0209] Among them, the trained direction discrimination network can fully learn the relationship between image direction and image spatial features, so as to accurately identify the image direction.
[0210] Among them, the training process of the direction discrimination network is similar to the training process of the existing network model. For the training process that is not described in detail, the training method of the existing network model can be referred to, and will not be repeated here.
[0211] The image direction of the j-th frame image and the image direction of the sample images in the training process mentioned above are only examples and can be adjusted according to actual application requirements without limitation thereto.
[0212] From the above content, it can be seen that by adjusting the directions of N frames of delivery images to a unified direction, the recognition efficiency and accuracy of subsequent images can be improved, thereby improving the accuracy and speed of delivery terminal delivery content detection.
[0213] Since the delivery video can be shot by passers-by (such as pedestrians, couriers, food delivery drivers), surveillance cameras, etc. facing the delivery terminal, the delivery video may be completely black or white, overexposed, or shoot the road and sky, etc.
[0214] In order to avoid this situation occurring in the delivered video on a large scale, which affects the subsequent image frame recognition, it is difficult to determine whether the image is: the delivered content is an image frame of the multimedia information to be delivered, thereby affecting the delivery content detection result.
[0215] To this end, in some embodiments of the present application, the delivery terminal delivery content detection method may further include: classifying the N frames of delivery images to determine the classification result of the delivery video.
[0216] The classification results include qualified videos and unqualified videos.
[0217] For example, first, each frame of the N frames of delivery images is input into the trained video discrimination network respectively, so that the preset video discrimination network classifies each frame of the image and determines whether each frame of the N frames of delivery images is qualified or unqualified.
[0218] Then, the number K of unqualified images in the N frames of delivered images is counted, and the proportion K / N of unqualified images in the N frames of delivered images is determined.
[0219] If the unqualified image ratio K / N is greater than the preset ratio threshold, the classification result of the delivered video is determined to be unqualified. If the unqualified image ratio K / N is less than or equal to the preset ratio threshold, the classification result of the delivered video is determined to be qualified.
[0220] Among them, the above-mentioned video discrimination network can be obtained in the following way: based on the training data set (including positive samples and negative samples, positive samples are correctly shot images, and negative samples are unqualified images such as all black and white, overexposed, and shooting the road sky), the preset video discrimination network is trained, so that the trained video discrimination network learns the characteristics of qualified images, thereby obtaining a trained video discrimination network (suitable for binary classification of images as qualified or unqualified).
[0221] The preset video discrimination network may be an open source network model that can be used for classification tasks, such as a YOLOv3 network, a MobileNet network, etc. Specifically, an open source network (that can be used for classification tasks) with model parameters as default values may be used as the preset video discrimination network.
[0222] At this time, correspondingly, the step of "performing content matching for each frame of the N frames of delivered images respectively to obtain a target image whose delivered content is consistent with the multimedia information to be delivered" in step S30 specifically includes: when the classification result is that the video is qualified, performing content matching for each frame of the N frames of delivered images respectively to obtain a target image whose delivered content is consistent with the multimedia information to be delivered.
[0223] From the above content, it can be seen that by classifying N frames of delivery images as qualified or unqualified to determine whether the delivery video is shot qualified, and then performing further data processing, it is possible to avoid the problem of low detection accuracy of the delivery content of the delivery terminal due to processing of unqualified delivery videos, thereby improving the detection accuracy of the delivery content of the delivery terminal to a certain extent.
[0224] In order to better implement the delivery terminal delivery content detection method in the embodiment of the present application, based on the delivery terminal delivery content detection method, the embodiment of the present application also provides a delivery terminal delivery content detection device, such as Figure 7 As shown, it is a schematic diagram of the structure of an embodiment of a delivery terminal delivery content detection device in an embodiment of the present application, and the delivery terminal delivery content detection device 700 includes:
[0225] The acquisition unit 701 is used to acquire the delivery video of the delivery terminal;
[0226] The interception unit 702 is used to intercept the image frames of the delivery video according to a preset interception frequency to obtain N frames of delivery images of the delivery terminal, wherein each frame of the N frames of delivery images has a frame identification number, and the frame identification number is used to indicate the delivery time sequence of the N frames of delivery images;
[0227] The matching unit 703 is used to perform content matching on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered;
[0228] The determination unit 704 is used to determine the delivery duration of the multimedia information according to the frame identification number, the target image and the interception frequency.
[0229] In some embodiments of the present application, the matching unit 703 is specifically used to:
[0230] Acquire reference text information of the multimedia information;
[0231] Performing text detection processing on each frame of image;
[0232] When target text information of each frame image is detected, each frame image is used as the target image, wherein the target text information refers to text information in the text information of each frame image that is the same as the reference text information.
[0233] In some embodiments of the present application, the matching unit 703 is specifically used to:
[0234] Acquire the similarity between each frame image and a preset reference image, wherein the reference image matches the multimedia information;
[0235] When the similarity is greater than a preset similarity threshold, each frame of the image is used as the target image.
[0236] In some embodiments of the present application, the determining unit 704 is specifically configured to:
[0237] Obtaining a target frame identification number of the target image;
[0238] Determining the start frame and the end frame of the continuous delivery of the multimedia information according to the target frame identification number;
[0239] The delivery duration of the multimedia information is determined according to the start frame, the end frame and the interception frequency.
[0240] In some embodiments of the present application, the number of the start frames and the number of the end frames are both x, where x is a positive integer greater than 1, and the determining unit 704 is specifically configured to:
[0241] The image frames between the i-th start frame and the i-th end frame are used as the continuous delivery segments of the multimedia information, wherein the value of i ranges from 1 to x;
[0242] Obtain the first image frame number between the (i-1)th end frame and the i-th start frame;
[0243] When the first image frame number is less than a preset frame number threshold, updating the continuous delivery segment according to the image frames between the (i-1)th start frame and the i-th end frame;
[0244] Obtaining an updated second image frame number of the continuously delivered segment;
[0245] The delivery duration of the multimedia information is determined according to the second image frame number and the interception frequency.
[0246] In some embodiments of the present application, the total duration of the delivered video is a first duration, and the delivery terminal delivery content detection device 700 further includes a statistical unit (not shown in the figure), and the statistical unit is specifically used to:
[0247] The updated number of the continuous delivery segments is counted as the delivery number of the multimedia information within the first duration.
[0248] In some embodiments of the present application, the determining unit 704 is specifically configured to:
[0249] Determining a second duration of the continuous delivery segment according to the second image frame number and the interception frequency;
[0250] The delivery duration of the multimedia information is counted according to the second duration.
[0251] In some embodiments of the present application, the matching unit 703 is specifically used to:
[0252] Extracting a static delivery area and a dynamic delivery area in each frame of the image;
[0253] Performing text detection processing on the static delivery area;
[0254] When it is detected that there is a first text in the static delivery area, each frame of the image is used as the target image, wherein the first text refers to a text that is the same as a preset reference static text;
[0255] In some embodiments of the present application, the delivery terminal delivery content detection device 700 further includes a discarding unit (not shown in the figure), and the discarding unit is specifically used to:
[0256] Performing text detection processing on the dynamic delivery areas of the image frames in the continuous delivery segments respectively to obtain a second text of the continuous delivery segments, wherein the second text is the same as a preset reference dynamic text;
[0257] When the amount of the second texts is less than a preset amount threshold, the continuous delivery segments are discarded.
[0258] In some embodiments of the present application, the delivery terminal delivery content detection device 700 further includes a direction adjustment unit (not shown in the figure). Before the step of performing content matching on each frame of the N delivery images to obtain a target image whose delivery content is consistent with the multimedia information to be delivered, the direction adjustment unit is specifically used to:
[0259] The directions of the N frames of projected images are adjusted to a unified direction.
[0260] In some embodiments of the present application, after the step of intercepting the image frames of the delivery video according to the preset interception frequency to obtain N frames of delivery images of the delivery terminal, the delivery terminal delivery content detection device 700 further includes a determination unit (not shown in the figure), which is specifically used to:
[0261] Classification processing is performed on the N frames of delivery images to determine a classification result of the delivery video, wherein the classification result includes that the video is qualified.
[0262] In some embodiments of the present application, the matching unit 703 is specifically used to:
[0263] When the classification result indicates that the video is qualified, content matching is performed on each of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered.
[0264] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0265] Since the delivery terminal delivery content detection device can execute the present application as follows Figures 1 to 6 Corresponding to the steps in the method for detecting the content delivered by the delivery terminal in any embodiment, the present application can be implemented as follows Figures 1 to 6 The beneficial effects that can be achieved by the method for detecting content delivered by a delivery terminal in any embodiment are detailed in the previous description and will not be repeated here.
[0266] In addition, in order to better implement the method for detecting the content delivered by the delivery terminal in the embodiment of the present application, on the basis of the method for detecting the content delivered by the delivery terminal, the embodiment of the present application also provides an electronic device, referring to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. Specifically, the electronic device provided by the embodiment of the present application includes a processor 801, and the processor 801 is used to execute a computer program stored in a memory 802 to implement the following Figures 1 to 6 Corresponding to each step of the method for detecting content delivered by a delivery terminal in any embodiment; or, the processor 801 is used to execute the computer program stored in the memory 802 to implement the following Figure 7 The functions of each unit in the corresponding embodiment.
[0267] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 802, and executed by the processor 801 to complete the embodiment of the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.
[0268] The electronic device may include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art will appreciate that the illustration is merely an example of an electronic device and does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the illustration, or may combine certain components, or different components, for example, the electronic device may also include input and output devices, network access devices, buses, etc., and the processor 801, memory 802, input and output devices, and network access devices, etc., are connected via a bus.
[0269] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.
[0270] The memory 802 can be used to store computer programs and / or modules. The processor 801 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 802 and calling the data stored in the memory 802. The memory 802 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, video data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0271] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described delivery terminal delivery content detection device, electronic device and its corresponding units can refer to the following. Figures 1 to 6 The description of the method for detecting the content delivered by the delivery terminal in any embodiment will not be repeated here in detail.
[0272] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0273] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the present application as follows: Figures 1 to 6 Corresponding to the steps in the method for detecting the content delivered by the delivery terminal in any embodiment, the specific operations can be referred to as follows Figures 1 to 6 The description of the method for detecting the content delivered by the delivery terminal in any embodiment will not be repeated here.
[0274] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0275] Due to the instructions stored in the computer-readable storage medium, the present application can be executed. Figures 1 to 6 Corresponding to the steps in the method for detecting the content delivered by the delivery terminal in any embodiment, the present application can be implemented as follows Figures 1 to 6The beneficial effects that can be achieved by the method for detecting content delivered by a delivery terminal in any embodiment are detailed in the previous description and will not be repeated here.
[0276] The above is a detailed introduction to a method, device, electronic device and computer-readable storage medium for detecting content delivered by a delivery terminal provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A method for detecting content delivered by a delivery terminal, characterized in that: The method comprises: Get the delivery video of the delivery terminal; According to a preset interception frequency, the image frames of the delivery video are intercepted to obtain N frames of delivery images of the delivery terminal, wherein each frame of the N frames of delivery images is provided with a frame identification number, and the frame identification number is used to indicate the delivery time sequence of the N frames of delivery images; Performing content matching on each of the N frames of delivered images to obtain a target image whose delivered content matches the multimedia information to be delivered; Obtaining a target frame identification number of the target image; Determining the start frame and the end frame of the continuous delivery of the multimedia information according to the target frame identification number; Determining a delivery duration of the multimedia information according to the start frame, the end frame and the interception frequency; The number of the start frames and the number of the end frames are both x, where x is a positive integer greater than 1, and determining the delivery duration of the multimedia information according to the start frame, the end frame and the interception frequency includes: The image frames between the i-th start frame and the i-th end frame are used as the continuous delivery segments of the multimedia information, wherein the value of i ranges from 1 to x; Obtain the first image frame number between the (i-1)th end frame and the i-th start frame; When the first image frame number is less than a preset frame number threshold, updating the continuous delivery segment according to the image frames between the (i-1)th start frame and the i-th end frame; Obtaining an updated second image frame number of the continuously delivered segment; The delivery duration of the multimedia information is determined according to the second image frame number and the interception frequency.
2. The method for detecting the content delivered by a delivery terminal according to claim 1, characterized in that: The performing content matching on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered includes: Acquire reference text information of the multimedia information; Performing text detection processing on each frame of image; When target text information of each frame image is detected, each frame image is used as the target image, wherein the target text information refers to text information in the text information of each frame image that is the same as the reference text information.
3. The method for detecting the content delivered by a delivery terminal according to claim 1, characterized in that: The performing content matching on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered includes: Acquire the similarity between each frame image and a preset reference image, wherein the reference image matches the multimedia information; When the similarity is greater than a preset similarity threshold, each frame of the image is used as the target image.
4. The method for detecting the content delivered by a delivery terminal according to claim 1, characterized in that: The total duration of the delivered video is a first duration, and the method further includes: The updated number of the continuous delivery segments is counted as the delivery number of the multimedia information within the first duration.
5. The method for detecting the content delivered by a delivery terminal according to claim 1, characterized in that: The determining, according to the second image frame number and the interception frequency, the delivery duration of the multimedia information includes: Determining a second duration of the continuous delivery segment according to the second image frame number and the interception frequency; The delivery duration of the multimedia information is counted according to the second duration.
6. The method for detecting the content delivered by a delivery terminal according to claim 1, characterized in that: The performing content matching on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered includes: Extracting a static delivery area and a dynamic delivery area in each frame of the image; Performing text detection processing on the static delivery area; When it is detected that there is a first text in the static delivery area, each frame of the image is used as the target image, wherein the first text refers to a text that is the same as a preset reference static text; The method further comprises: Performing text detection processing on the dynamic delivery areas of the image frames in the continuous delivery segments respectively to obtain a second text of the continuous delivery segments, wherein the second text is the same as a preset reference dynamic text; When the amount of the second texts is less than a preset amount threshold, the continuous delivery segments are discarded.
7. The method for detecting content delivered by a delivery terminal according to any one of claims 1 to 6, characterized in that: The content matching is performed on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered, and the process also includes: The directions of the N frames of projected images are adjusted to a unified direction.
8. The method for detecting content delivered by a delivery terminal according to any one of claims 1 to 6, characterized in that: The method further includes: intercepting image frames of the delivery video according to a preset interception frequency to obtain N frames of delivery images of the delivery terminal; and then: Classify the N frames of delivery images to determine a classification result of the delivery video, wherein the classification result includes that the video is qualified; The performing content matching on each frame of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered includes: When the classification result indicates that the video is qualified, content matching is performed on each of the N frames of delivered images to obtain a target image whose delivered content is consistent with the multimedia information to be delivered.
9. A device for detecting content delivered by a delivery terminal, characterized in that: The delivery terminal delivery content detection device comprises: An acquisition unit, used for acquiring the delivery video of the delivery terminal; An interception unit, used to intercept the image frames of the delivery video according to a preset interception frequency to obtain N frames of delivery images of the delivery terminal, wherein each frame of the N frames of delivery images has a frame identification number, and the frame identification number is used to indicate the delivery time sequence of the N frames of delivery images; A matching unit, configured to perform content matching on each of the N frames of delivered images to obtain a target image whose delivered content matches the multimedia information to be delivered; Identify units for: Obtaining a target frame identification number of the target image; Determining the start frame and the end frame of the continuous delivery of the multimedia information according to the target frame identification number; Determining a delivery duration of the multimedia information according to the start frame, the end frame and the interception frequency; The number of the start frames and the number of the end frames are both x, where x is a positive integer greater than 1, and determining the delivery duration of the multimedia information according to the start frame, the end frame and the interception frequency includes: The image frames between the i-th start frame and the i-th end frame are used as the continuous delivery segments of the multimedia information, wherein the value of i ranges from 1 to x; Obtain the first image frame number between the (i-1)th end frame and the i-th start frame; When the first image frame number is less than a preset frame number threshold, updating the continuous delivery segment according to the image frames between the (i-1)th start frame and the i-th end frame; Obtaining an updated second image frame number of the continuously delivered segment; The delivery duration of the multimedia information is determined according to the second image frame number and the interception frequency.
10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the method for detecting content delivered by a delivery terminal as claimed in any one of claims 1 to 8 is executed.
11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method for detecting content delivered by a delivery terminal as described in any one of claims 1 to 8.
Citation Information
Patent Citations
Video analysis method and device
CN110532833A
Advertisement putting detection method, device, electronic equipment and storage medium
CN110992092A
Target detection method, device and system, and computer readable storage medium
CN114092845A
