Method and apparatus for outputting information

Abstract

The embodiments of the present application disclose a method and device for outputting information. A specific implementation of the method includes: obtaining an original video clip showing a courier sorting process; performing preset processing on the original video clip to generate at least one candidate video clip; for each candidate video clip in at least one candidate video clip, based on the candidate video clip and a pre-trained target sorting action recognition model, determining a target sorting action score corresponding to the candidate video clip, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to characterize the probability that the candidate video clip presents a target sorting action; based on the target sorting action score corresponding to at least one candidate video clip, selecting a target video clip from at least one candidate video clip, and outputting the video information of the target video clip. This implementation improves the efficiency and accuracy of target sorting action recognition.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method and device for outputting information. Background Art

[0002] At present, in the logistics operation system, there are a large number of compensation losses every month, of which the compensation caused by damaged goods accounts for a large proportion, and most of them are caused by violent operations. At present, violent operations in the logistics sorting process are usually identified manually, which takes up a lot of human resources and is difficult to monitor in real time. The use of artificial intelligence video processing technology can solve this problem well. Summary of the invention

[0003] The embodiments of the present application provide a method and device for outputting information.

[0004] In a first aspect, an embodiment of the present application provides a method for outputting information, comprising: obtaining an original video clip presenting a courier sorting process; performing preset processing on the original video clip to generate at least one candidate video clip; for each candidate video clip in the at least one candidate video clip, based on the candidate video clip and a pre-trained target sorting action recognition model, determining a target sorting action score corresponding to the candidate video clip, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to characterize the probability that the candidate video clip presents a target sorting action; based on the target sorting action score corresponding to the at least one candidate video clip, selecting a target video clip from the at least one candidate video clip, and outputting video information of the target video clip.

[0005] In some embodiments, the original video clip is subjected to preset processing to generate at least one candidate video clip, including: detecting image frames in the original video clip, deleting image frames in the original video clip that have preset abnormal conditions; cropping the image frames in the video clip after the image frames are deleted into a preset size; dividing the cropped video clip according to preset duration to obtain at least one candidate video clip.

[0006] In some embodiments, based on the candidate video segment and a pre-trained target sorting action recognition model, determining a target sorting action score corresponding to the candidate video segment includes: dividing the candidate video segment into at least one candidate video sub-segment, wherein the candidate video sub-segment includes a continuous preset first number of image frames; for each candidate video sub-segment in the at least one candidate video sub-segment, inputting the candidate video sub-segment into the pre-trained target sorting action recognition model to obtain a target sorting action score corresponding to the candidate video sub-segment; determining an average value of at least one target sorting action score corresponding to the at least one candidate video sub-segment, and determining the average value as the target sorting action score corresponding to the candidate video segment.

[0007] In some embodiments, based on the target sorting action score corresponding to at least one candidate video segment, a target video segment is selected from at least one candidate video segment, including: generating a time domain waveform for representing the change of the target sorting action score with the time of the corresponding at least one candidate video segment, and performing curve smoothing on the time domain waveform; based on at least one target sorting action score after the curve smoothing, the target video segment is selected from at least one candidate video segment.

[0008] In some embodiments, based on at least one target sorting action score after curve smoothing processing, selecting a target video segment from at least one candidate video segment includes: selecting a candidate video segment whose corresponding target sorting action score after curve smoothing processing is greater than a preset score threshold from at least one candidate video segment; and selecting a preset second number of video segments from the candidate video segments greater than the score threshold in descending order of the corresponding target sorting action score after curve smoothing processing as the target video segments.

[0009] In some embodiments, the video information includes a time period corresponding to the video.

[0010] In a second aspect, an embodiment of the present application provides a device for outputting information, comprising: an acquisition unit, configured to acquire an original video clip presenting a courier sorting process; a generation unit, configured to perform preset processing on the original video clip to generate at least one candidate video clip; a determination unit, configured to determine, for each candidate video clip in the at least one candidate video clip, a target sorting action score corresponding to the candidate video clip based on the candidate video clip and a pre-trained target sorting action recognition model, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to characterize the probability that the candidate video clip presents a target sorting action; an output unit, configured to select a target video clip from the at least one candidate video clip based on the target sorting action score corresponding to the at least one candidate video clip, and output video information of the target video clip.

[0011] In some embodiments, the generation unit is further configured to perform preset processing on the original video clip in the following manner to generate at least one candidate video clip: detect image frames in the original video clip, delete image frames in the original video clip that have preset abnormal conditions; crop the image frames in the video clip after deleting the image frames into a preset size; divide the cropped video clip according to preset time length to obtain at least one candidate video clip.

[0012] In some embodiments, the determination unit is further configured to determine the target sorting action score corresponding to the candidate video segment based on the candidate video segment and a pre-trained target sorting action recognition model in the following manner: dividing the candidate video segment into at least one candidate video sub-segment, wherein the candidate video sub-segment includes a continuous preset first number of image frames; for each candidate video sub-segment in the at least one candidate video sub-segment, inputting the candidate video sub-segment into the pre-trained target sorting action recognition model to obtain the target sorting action score corresponding to the candidate video sub-segment; determining an average value of at least one target sorting action score corresponding to at least one candidate video sub-segment, and determining the average value as the target sorting action score corresponding to the candidate video segment.

[0013] In some embodiments, the output unit is further configured to select a target video segment from at least one candidate video segment based on the target sorting action score corresponding to the at least one candidate video segment in the following manner: generate a time domain waveform for characterizing the change of the target sorting action score with the time of the corresponding at least one candidate video segment, and perform curve smoothing on the time domain waveform; and select a target video segment from at least one candidate video segment based on at least one target sorting action score after curve smoothing.

[0014] In some embodiments, the output unit is further configured to select a target video segment from at least one candidate video segment based on at least one target sorting action score after curve smoothing processing in the following manner: select a candidate video segment whose corresponding target sorting action score after curve smoothing processing is greater than a preset score threshold from at least one candidate video segment; and select a preset second number of video segments as target video segments from the candidate video segments greater than the score threshold in descending order of the corresponding target sorting action score after curve smoothing processing.

[0015] In some embodiments, the video information includes a time period corresponding to the video.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation method in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0018] The method and device for outputting information provided by the above-mentioned embodiments of the present application obtain an original video clip showing the express sorting process; then, the above-mentioned original video clip is preset to generate at least one candidate video clip; then, for each candidate video clip in the above-mentioned at least one candidate video clip, based on the candidate video clip and the pre-trained target sorting action recognition model, the target sorting action score corresponding to the candidate video clip is determined; finally, based on the target sorting action score corresponding to the above-mentioned at least one candidate video clip, a target video clip is selected from the above-mentioned at least one candidate video clip, and the video information of the above-mentioned target video clip is output. Compared with manual recognition of target sorting actions, this method improves the recognition efficiency of target sorting actions. At the same time, the use of a three-dimensional convolutional neural network can well capture the target sorting action in the video clip, thereby accurately locating the moment of the target sorting action (violent sorting action), which improves the accuracy of target sorting action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0020] Figure 1 is an exemplary system architecture diagram to which various embodiments of the present application may be applied;

[0021] Figure 2 is a flow chart of an embodiment of a method for outputting information according to the present application;

[0022] Figure 3 is a schematic diagram of an application scenario of the method for outputting information according to the present application;

[0023] Figure 4 is a flow chart of another embodiment of a method for outputting information according to the present application;

[0024] Figure 5 is a schematic structural diagram of an embodiment of a device for outputting information according to the present application;

[0025] Figure 6 It is a structural diagram of a computer system suitable for implementing an electronic device of an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the method for outputting information of the present application may be applied.

[0029] like Figure 1 As shown, the system architecture 100 may include a camera 101, networks 1021, 1022, a server 103, and output terminals 1041, 1042, 1043. The network 1021 is used to provide a medium for a communication link between the camera 101 and the server 103. The network 1022 is used to provide a medium for a communication link between the server 103 and the output terminals 1041, 1042, 1043. The networks 1021, 1022 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0030] The camera device 101 can interact with the server 103 via the network 1021 to send or receive messages (for example, the server 103 can obtain the original video clip showing the express sorting process from the camera device 101).

[0031] The output terminals 1041, 1042, 1043 can interact with the server 103 through the network 1022 to send or receive messages, etc. (for example, the output terminals 1041, 1042, 1043 can receive video information of the target video clip output by the server 103). Various communication client applications can be installed on the output terminals 1041, 1042, 1043, such as video playback applications, logistics management applications, instant messaging software, etc.

[0032] The output terminals 1041, 1042, 1043 may be hardware or software. When the output terminals 1041, 1042, 1043 are hardware, they may be various electronic devices that support information interaction, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc. When the output terminals 1041, 1042, 1043 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.

[0033] The server 103 may be a server that provides various services. For example, it may be a background server that analyzes the original video clip showing the express sorting process. The server 103 may first obtain the original video clip showing the express sorting process from the camera device 101; then, the original video clip may be pre-processed to generate at least one candidate video clip; then, for each candidate video clip in the at least one candidate video clip, based on the candidate video clip and the pre-trained target sorting action recognition model, the target sorting action score corresponding to the candidate video clip may be determined; finally, based on the target sorting action score corresponding to the at least one candidate video clip, a target video clip may be selected from the at least one candidate video clip, and the video information of the target video clip may be output, for example, the video information of the target video clip may be output to the output terminals 1041, 1042, and 1043.

[0034] It should be noted that the server 103 can be hardware or software. When the server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 103 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0035] It should be noted that the method for outputting information provided in the embodiment of the present application is usually executed by the server 103.

[0036] It should be noted that the server 103 may store the original video clips showing the express sorting process locally, and the server 103 may obtain the original video clips locally. In this case, the exemplary system architecture 100 may not have the camera device 101 and the network 1021.

[0037] It should also be noted that the server 103 may be connected to a display device (eg, a display screen) to display the video information of the output target video segment. In this case, the exemplary system architecture 100 may not have the network 1022 and the output terminals 1041, 1042, 1043.

[0038] It should be understood that Figure 1 The number of camera devices, networks, servers and output terminals in the embodiment is only for illustration. Any number of camera devices, networks, servers and output terminals may be provided according to the implementation requirements.

[0039] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for outputting information according to the present application. The method for outputting information comprises the following steps:

[0040] Step 201, obtaining an original video clip showing the express sorting process.

[0041] In this embodiment, the execution subject of the method for outputting information (for example Figure 1 The server shown in the figure can obtain the original video clip showing the express sorting process. Express sorting generally refers to the process in which the logistics distribution center quickly and accurately picks the goods from their storage locations or other locations according to the customer's order requirements or distribution plan, and classifies and concentrates them in a certain way. The above-mentioned execution entity can obtain the above-mentioned original video clip from the camera device installed in the logistics distribution center. The above-mentioned camera device can capture the sorting process of the sorter sorting the express to obtain the original video clip.

[0042] Here, the logistics distribution center can be installed with multiple cameras, each of which can shoot the express sorting process from different angles to obtain original video clips.

[0043] Step 202: Perform a preset process on the original video segment to generate at least one candidate video segment.

[0044] In this embodiment, the execution subject may perform a preset process on the original video segment obtained in step 201 to generate at least one candidate video segment. As an example, the execution subject may divide the original video segment according to a preset duration (e.g., 2 minutes) to obtain at least one candidate video segment.

[0045] Step 203: for each candidate video segment in at least one candidate video segment, based on the candidate video segment and a pre-trained target sorting action recognition model, determine a target sorting action score corresponding to the candidate video segment.

[0046] In this embodiment, for each candidate video segment in the at least one candidate video segment, the execution subject may determine the target sorting action score corresponding to the candidate video segment based on the candidate video segment and the pre-trained target sorting action recognition model. Here, the target sorting action may include a violent sorting action. If the distance between the express package and the sorter's hand exceeds 15 cm before the express package lands, the sorter's current action may be defined as a violent sorting action.

[0047] Specifically, the above-mentioned execution subject can input the candidate video clip into the above-mentioned target sorting action recognition model to obtain the target sorting action score corresponding to the candidate video clip. The above-mentioned target sorting action score can be used to characterize the probability that the candidate video clip presents a target sorting action. The above-mentioned target sorting action recognition model can be used to characterize the video clip and the target sorting action score corresponding to the video clip. Here, the above-mentioned target sorting action recognition model may include a three-dimensional convolutional neural network (3D CNN, 3D Convolutional Neural Networks), for example, P3DResNet (Pseudo-3D Residual Networks, pseudo 3D residual network). The three-dimensional convolutional neural network is mainly used in the fields of video classification and action recognition. It is changed on the basis of the two-dimensional convolutional neural network. Since the two-dimensional convolutional neural network cannot capture the time sequence information well, we use a three-dimensional convolutional neural network, so that the time sequence information in the video can be well utilized. In the three-dimensional convolutional neural network, the time dimension can be regarded as the third dimension. The 3D convolution is formed by stacking multiple continuous frames into a cube, and then applying the 3D convolution kernel in the cube. In this structure, each feature map in the convolutional layer is connected to multiple adjacent consecutive frames in the previous layer, thus capturing motion information.

[0048] It should be noted that if the target sorting action is a violent sorting action, the target sorting action recognition model is usually a violent sorting action recognition model, and the target sorting action score is usually a violent sorting action score, which is used to characterize the probability that the candidate video clip presents a violent sorting action.

[0049] Here, we can obtain image and video classification data from academic datasets such as ImageNet, UCV-101, and Sports-1M, and then collect certain training data for the target sorting actions in logistics scenarios and have them annotated by specific referees. Then, we use the training set to pre-train the initial model and obtain the target sorting action recognition model. The training set includes sample video clips and the target sorting action scores corresponding to the sample video clips.

[0050] Step 204 : based on the target sorting action score corresponding to the at least one candidate video segment, a target video segment is selected from the at least one candidate video segment, and video information of the target video segment is output.

[0051] In this embodiment, the execution subject may select a target video segment from the at least one candidate video segment based on the target sorting action score corresponding to the at least one candidate video segment determined in step 203. Specifically, the execution subject may select a candidate video segment having the highest corresponding target sorting action score from the at least one candidate video segment as the target video segment.

[0052] Afterwards, the execution subject may output the video information of the target video segment. The video information may include but is not limited to at least one of the following: the target video segment and the target sorting action score corresponding to the target video segment.

[0053] Compared with manual recognition of target sorting actions, the method provided by the above embodiment of the present application improves the recognition efficiency of target sorting actions. At the same time, the use of a three-dimensional convolutional neural network can well capture the target sorting action in the video clip, thereby accurately locating the moment of the target sorting action (violent sorting action), which improves the accuracy of target sorting action recognition.

[0054] In some optional implementations of the present embodiment, the execution subject may perform preset processing on the original video clip to generate at least one candidate video clip in the following manner: the execution subject may first detect the image frames in the original video clip, and delete the image frames in the original video clip that have preset abnormal conditions. Here, the image frames in the abnormal conditions may include grayscale images. In this case, the execution subject may perform grayscale image detection on the original video clip. If the original video clip contains a grayscale image, the detected grayscale image may be deleted. The image frames in the abnormal conditions may also include a flower screen image. In this case, the execution subject may perform flower screen detection on the original video clip. If there is a flower screen portion in the original video clip, the detected flower screen portion may be deleted. Afterwards, the execution subject may crop the image frames in the video clip after deleting the image frames into a preset size. The above-mentioned size is usually related to the size of the input image required by the target sorting action recognition model for identifying the target sorting action in the video clip. For example, if the target sorting action recognition model requires the input image size to be 256×256, the above-mentioned execution subject can crop the image frames in the video clip after deleting the image frames into an image size of 256×256. Finally, the above-mentioned execution subject can divide the cropped video clip according to the preset time length to obtain at least one candidate video clip. For example, the cropped video clip can be divided according to the time length of 10 minutes. In this way, abnormal image frames in the video can be deleted, thereby improving the accuracy of identifying the target action from the processed image. At the same time, cropping the size of the image frame can make the cropped image frame more suitable for model processing. In this way, images with poor image quality can be eliminated, and image frames without image abnormalities can be used for target sorting action recognition, which can further improve the accuracy of target sorting action recognition.

[0055] In some optional implementations of this embodiment, the execution subject may determine the target sorting action score corresponding to the candidate video segment based on the candidate video segment and the pre-trained target sorting action recognition model in the following manner: the execution subject may divide the candidate video segment into at least one candidate video sub-segment. Here, each candidate video sub-segment in the at least one candidate video sub-segment may include a continuous preset first number of image frames. As an example, if the candidate video segment is a 2-second video segment, the 2-second video segment may be divided into at least one video sub-segment in units of 16 consecutive frames with a step size of 1 or 2. Afterwards, for each candidate video sub-segment in the at least one candidate video sub-segment, the execution subject may input the candidate video sub-segment into the pre-trained target sorting action recognition model to obtain the target sorting action score corresponding to the candidate video sub-segment. The target sorting action recognition model may be used to characterize the video segment and the target sorting action score corresponding to the video segment. Here, the target sorting action recognition model may include a three-dimensional convolutional neural network, such as P3DResNet. Finally, the execution subject may determine the average value of at least one target sorting action score corresponding to the at least one candidate video sub-segment, and determine the average value as the target sorting action score corresponding to the candidate video segment. As an example, if the candidate video segment is divided into three candidate video sub-segments, and the target sorting action scores corresponding to the three candidate video sub-segments are 0.3, 0.3, and 0.4, respectively, the score of the candidate video segment may be 0.33. By fusing the target sorting action scores corresponding to the divided video segments in this way, the target sorting action scores corresponding to the candidate video segments obtained can be made more accurate, further improving the accuracy of target sorting action recognition.

[0056] In some optional implementations of this embodiment, the video information may include a time period corresponding to the video. The time period is usually composed of a start time and an end time, that is, the execution subject may output the start time and end time of the target video segment in the original video segment. As an example, the video information may include a video start time of 16 seconds and a video end time of 18 seconds.

[0057] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the method for outputting information according to this embodiment. Figure 3In the application scenario, the server 301 may first obtain the original video clip 303 showing the express sorting process from the camera device 302. Here, the camera device 302 is usually installed in a logistics distribution center to shoot the express sorting process. Afterwards, the server 301 may divide the original video clip 303 according to a duration of 2 minutes to obtain multiple candidate video clips. Here, the multiple candidate video clips are shown in icon 304. Taking the candidate video clip 3041 as an example, the server 301 may divide the candidate video clip 3041 into three candidate video sub-segments, and the three candidate video sub-segments may be respectively input into the three-dimensional convolutional neural network 305 to obtain the corresponding target sorting action scores 306. Here, the three-dimensional convolutional neural network 305 may be P3DResNet, and the target sorting action scores 306 corresponding to the three candidate video sub-segments obtained may be 0.3, 0.3 and 0.4 respectively. Then, the server 301 can determine the average value of the target sorting action scores 306 corresponding to the three candidate video sub-segments, 0.33, as the target sorting action score 3071 corresponding to the candidate video segment 3041. Similarly, it can be determined that the target sorting action score 3072 corresponding to the candidate video segment 3042 is 0.5, the target sorting action score 3073 corresponding to the candidate video segment 3043 is 0.8, and the target sorting action score 3074 corresponding to the candidate video segment 3044 is 0.37. Afterwards, the server 301 can select the candidate video segment corresponding to the highest target sorting action score from the multiple candidate video segments 304. Here, the highest target sorting action score 0.8 corresponds to the candidate video segment 3043, and the server 301 can determine the candidate video segment 3043 as the target video segment 308. Finally, the server 301 can output the video information 309 of the target video segment 308. Here, the video information 309 may be sent to the terminal device 310 for output, and the video information 309 may include the target video segment and the target sorting action score 0.8 corresponding to the target video segment.

[0058] Further references Figure 4 , which shows a process 400 of another embodiment of a method for outputting information according to the present application, comprising the following steps:

[0059] Step 401, obtaining an original video clip showing the express sorting process.

[0060] Step 402: Perform a preset process on the original video segment to generate at least one candidate video segment.

[0061] Step 403: for each candidate video segment in the at least one candidate video segment, based on the candidate video segment and a pre-trained target sorting action recognition model, determine a target sorting action score corresponding to the candidate video segment.

[0062] In this embodiment, the specific operations of steps 401-403 are already Figure 2 In the illustrated embodiment, steps 201 - 203 are described in detail and will not be repeated here.

[0063] Step 404: Generate a time domain waveform for representing the change of the target sorting action score with the time of the corresponding at least one candidate video segment, and perform curve smoothing processing on the time domain waveform.

[0064] In this embodiment, the execution subject of the method for outputting information (for example Figure 1 The server shown in the figure can generate a time domain waveform for representing the change of the target sorting action score with the time of the corresponding at least one candidate video segment. Specifically, the execution subject can use the time of the video segment as the horizontal axis value and the target sorting action score as the vertical axis value to draw the time domain waveform of the change of the target sorting action score with the time of the corresponding at least one candidate video segment. Here, the time of the candidate video segment can be the start time of the candidate video segment, the end time, or the midpoint time.

[0065] Afterwards, the above-mentioned execution subject can perform curve smoothing processing on the generated time domain waveform. Here, the generated time domain waveform can be smoothed by using methods such as quadratic exponential smoothing, least squares curve fitting and five-point cubic smoothing curve. It should be noted that curve smoothing methods such as quadratic exponential smoothing, least squares curve fitting and five-point cubic smoothing curve are well-known technologies that are currently widely studied and applied, and will not be repeated here.

[0066] Step 405 : Based on the score of at least one target sorting action after the curve smoothing process, a target video segment is selected from at least one candidate video segment, and video information of the target video segment is output.

[0067] In this embodiment, the execution subject may select the target video segment from the at least one candidate video segment based on the score of the at least one target sorting action after the curve smoothing process. Specifically, the execution subject may select the candidate video segment with the highest score of the target sorting action after the curve smoothing process from the at least one candidate video segment as the target video segment.

[0068] Afterwards, the execution subject may output the video information of the target video segment. The video information may include but is not limited to at least one of the following: the target video segment and the target sorting action score corresponding to the target video segment.

[0069] In some optional implementations of this embodiment, the execution subject may select the target video segment from the at least one candidate video segment based on the score of the at least one target sorting action after the curve smoothing process in the following manner: the execution subject may select the candidate video segment whose corresponding target sorting action score after the curve smoothing process is greater than a preset score threshold (for example, 0.6) from the at least one candidate video segment. Afterwards, a preset second number (for example, 3, 5) of video segments may be selected as the target video segment from the candidate video segments greater than the score threshold in descending order of the corresponding target sorting action score after the curve smoothing process.

[0070] from Figure 4 It can be seen that Figure 2 Compared with the corresponding embodiment, the process 400 of the method for outputting information in this embodiment embodies the steps of performing curve smoothing processing on the time domain waveform used to characterize the change of the target sorting action score with the time of the corresponding at least one candidate video segment, and selecting the video segment based on the target sorting action score after the curve smoothing processing. Therefore, the scheme described in this embodiment takes into account the temporal continuity of the target sorting action, and performing curve smoothing processing on the time domain waveform can avoid false detection and improve the accuracy of target sorting action recognition.

[0071] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for outputting information, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0072] like Figure 5As shown, the device 500 for outputting information in this embodiment includes: an acquisition unit 501, a generation unit 502, a determination unit 503 and an output unit 504. The acquisition unit 501 is configured to acquire an original video segment showing a courier sorting process; the generation unit 502 is configured to perform a preset process on the original video segment to generate at least one candidate video segment; the determination unit 503 is configured to determine, for each candidate video segment in at least one candidate video segment, a target sorting action score corresponding to the candidate video segment based on the candidate video segment and a pre-trained target sorting action recognition model, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to characterize the probability that the candidate video segment presents a target sorting action; the output unit 504 is configured to select a target video segment from at least one candidate video segment based on the target sorting action score corresponding to the at least one candidate video segment, and output the video information of the target video segment.

[0073] In some optional implementations of this embodiment, the generation unit 502 may perform preset processing on the original video clip to generate at least one candidate video clip in the following manner: the generation unit 502 may first detect the image frames in the original video clip and delete the image frames with preset abnormal conditions in the original video clip. Here, the image frames with abnormal conditions may include grayscale images. In this case, the generation unit 502 may perform grayscale image detection on the original video clip. If the original video clip contains a grayscale image, the detected grayscale image may be deleted. The image frames with abnormal conditions may also include a flower screen image. In this case, the generation unit 502 may perform flower screen detection on the original video clip. If there is a flower screen part in the original video clip, the detected flower screen part may be deleted. Afterwards, the generation unit 502 may crop the image frames in the video clip after deleting the image frames into a preset size. The above-mentioned size is usually related to the size of the input image required by the target sorting action recognition model for identifying the target sorting action in the video clip. For example, if the target sorting action recognition model requires the input image size to be 256×256, the above-mentioned generation unit 502 can crop the image frames in the video clip after deleting the image frames into an image size of 256×256. Finally, the above-mentioned generation unit 502 can divide the cropped video clip according to the preset time length to obtain at least one candidate video clip. For example, the cropped video clip can be divided according to the time length of 10 minutes. In this way, abnormal image frames in the video can be deleted, thereby improving the accuracy of identifying the target action from the processed image. At the same time, cropping the size of the image frame can make the cropped image frame more suitable for model processing. In this way, images with poor image quality can be eliminated, and image frames without image abnormalities can be used for target sorting action recognition, which can further improve the accuracy of target sorting action recognition.

[0074] In some optional implementations of this embodiment, the determination unit 503 may determine the target sorting action score corresponding to the candidate video segment based on the candidate video segment and the pre-trained target sorting action recognition model in the following manner: the determination unit 503 may divide the candidate video segment into at least one candidate video sub-segment. Here, each candidate video sub-segment in the at least one candidate video sub-segment may include a continuous preset first number of image frames. As an example, if the candidate video segment is a 2-second video segment, the 2-second video segment may be divided into at least one video sub-segment in units of 16 consecutive frames with a step size of 1 or 2. Afterwards, for each candidate video sub-segment in the at least one candidate video sub-segment, the determination unit 503 may input the candidate video sub-segment into the pre-trained target sorting action recognition model to obtain the target sorting action score corresponding to the candidate video sub-segment. The target sorting action recognition model may be used to characterize the video segment and the target sorting action score corresponding to the video segment. Here, the target sorting action recognition model may include a three-dimensional convolutional neural network, such as P3DResNet. Finally, the determination unit 503 may determine the average value of at least one target sorting action score corresponding to the at least one candidate video sub-segment, and determine the average value as the target sorting action score corresponding to the candidate video segment. As an example, if the candidate video segment is divided into three candidate video sub-segments, and the target sorting action scores corresponding to the three candidate video sub-segments are 0.3, 0.3, and 0.4, respectively, the score of the candidate video segment may be 0.33. By fusing the target sorting action scores corresponding to the divided video segments in this way, the target sorting action scores corresponding to the candidate video segments obtained may be more accurate, further improving the accuracy of target sorting action recognition.

[0075] In some optional implementations of this embodiment, the output unit 504 may select a target video segment from the at least one candidate video segment based on the target sorting action score corresponding to the at least one candidate video segment in the following manner: the output unit 504 may generate a time domain waveform for characterizing the change of the target sorting action score with the time of the corresponding at least one candidate video segment. Specifically, the output unit 504 may use the time of the video segment as the horizontal coordinate value and the target sorting action score as the vertical coordinate value to draw the time domain waveform of the change of the target sorting action score with the time of the corresponding at least one candidate video segment. Here, the time of the candidate video segment may be the start time, the end time, or the midpoint time of the candidate video segment. Afterwards, the output unit 504 may perform curve smoothing on the generated time domain waveform. Here, the generated time domain waveform may be smoothed by using quadratic exponential smoothing, least squares curve fitting, and five-point cubic smoothing curve. It should be noted that the curve smoothing methods such as quadratic exponential smoothing, least squares curve fitting, and five-point cubic smoothing curve are currently well-known technologies that are widely studied and applied, and will not be described in detail here.

[0076] In some optional implementations of this embodiment, the output unit 504 may select a target video segment from the at least one candidate video segment based on the at least one target sorting action score after the curve smoothing process in the following manner: the output unit 504 may select a candidate video segment whose corresponding target sorting action score after the curve smoothing process is greater than a preset score threshold (e.g., 0.6) from the at least one candidate video segment. Afterwards, a preset second number (e.g., 3, 5) of video segments may be selected as target video segments from the candidate video segments greater than the score threshold in descending order of the corresponding target sorting action scores after the curve smoothing process.

[0077] In some optional implementations of this embodiment, the video information may include a time period corresponding to the video. The time period is usually composed of a start time and an end time, that is, the execution subject may output the start time and end time of the target video segment in the original video segment. As an example, the video information may include a video start time of 16 seconds and a video end time of 18 seconds.

[0078] Reference below Figure 6 , which shows an electronic device (eg, Figure 1 A schematic diagram of the structure of the server in FIG600. Figure 6 The server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0079] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0080] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0081] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, an apparatus, or a device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, an apparatus, or a device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0082] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains an original video clip showing the express sorting process; performs preset processing on the original video clip to generate at least one candidate video clip; for each candidate video clip in the at least one candidate video clip, based on the candidate video clip and a pre-trained target sorting action recognition model, determines the target sorting action score corresponding to the candidate video clip, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to characterize the probability that the candidate video clip presents a target sorting action; based on the target sorting action score corresponding to the at least one candidate video clip, selects a target video clip from the at least one candidate video clip, and outputs the video information of the target video clip.

[0083] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0084] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor includes an acquisition unit, a generation unit, a determination unit, and an output unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring an original video clip showing a courier sorting process".

[0086] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims

1. A method for outputting information, comprising: Obtaining the original video clip showing the express sorting process; Divide the original video segment according to a preset duration to generate at least one candidate video segment, wherein the preset duration is in the order of minutes; For each candidate video segment in the at least one candidate video segment, based on the candidate video segment and a pre-trained target sorting action recognition model, determining a target sorting action score corresponding to the candidate video segment, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to represent the probability that the candidate video segment presents a target sorting action; generating a time domain waveform for representing a change in a target sorting action score with respect to the time of the corresponding at least one candidate video segment, and performing curve smoothing processing on the time domain waveform; Based on at least one target sorting action score after the curve smoothing process, a target video segment is selected from the at least one candidate video segment, and video information of the target video segment is output.

2. The method according to claim 1, wherein: The dividing the original video segment according to the preset duration to generate at least one candidate video segment includes: Detecting the image frames in the original video clip, and deleting the image frames in the original video clip that have preset abnormal conditions; cropping the image frames in the video clip after deleting the image frames into a preset size; The cropped video segment is divided according to a preset duration to obtain at least one candidate video segment.

3. The method according to claim 1, wherein: The step of determining the target sorting action score corresponding to the candidate video segment based on the candidate video segment and a pre-trained target sorting action recognition model includes: Dividing the candidate video segment into at least one candidate video sub-segment, wherein the candidate video sub-segment includes a continuous preset first number of image frames; For each candidate video sub-segment in the at least one candidate video sub-segment, input the candidate video sub-segment into a pre-trained target sorting action recognition model to obtain a target sorting action score corresponding to the candidate video sub-segment; An average value of at least one target sorting action score corresponding to the at least one candidate video sub-segment is determined, and the average value is determined as the target sorting action score corresponding to the candidate video segment.

4. The method according to claim 1, wherein: The step of selecting a target video segment from the at least one candidate video segment based on the at least one target sorting action score after the curve smoothing process comprises: Selecting, from the at least one candidate video segment, a candidate video segment whose corresponding target sorting action score after curve smoothing is greater than a preset score threshold; A preset second number of video segments are selected as target video segments from the candidate video segments having scores greater than the score threshold in descending order of the target sorting action scores after the corresponding curve smoothing processing.

5. The method according to any one of claims 1 to 4, wherein: The video information includes a time period corresponding to the video.

6. A device for outputting information, comprising: An acquisition unit is configured to acquire an original video clip showing a courier sorting process; A generating unit is configured to divide the original video segment according to a preset duration to generate at least one candidate video segment, wherein the preset duration is in the order of minutes; a determining unit configured to determine, for each candidate video segment in the at least one candidate video segment, a target sorting action score corresponding to the candidate video segment based on the candidate video segment and a pre-trained target sorting action recognition model, wherein the target sorting action recognition model includes a three-dimensional convolutional neural network, and the target sorting action score is used to represent the probability that the candidate video segment presents a target sorting action; The output unit is configured to generate a time domain waveform for representing the change of the target sorting action score with the time of the corresponding at least one candidate video segment, and perform curve smoothing processing on the time domain waveform; based on at least one target sorting action score after the curve smoothing processing, select a target video segment from the at least one candidate video segment, and output the video information of the target video segment.

7. The device according to claim 6, wherein: The generating unit is further configured to divide the original video segment according to a preset duration in the following manner to generate at least one candidate video segment: Detecting the image frames in the original video clip, and deleting the image frames in the original video clip that have preset abnormal conditions; cropping the image frames in the video clip after deleting the image frames into a preset size; The cropped video segment is divided according to a preset duration to obtain at least one candidate video segment.

8. The device according to claim 6, wherein: The determining unit is further configured to determine the target sorting action score corresponding to the candidate video segment based on the candidate video segment and the pre-trained target sorting action recognition model in the following manner: Dividing the candidate video segment into at least one candidate video sub-segment, wherein the candidate video sub-segment includes a continuous preset first number of image frames; For each candidate video sub-segment in the at least one candidate video sub-segment, input the candidate video sub-segment into a pre-trained target sorting action recognition model to obtain a target sorting action score corresponding to the candidate video sub-segment; An average value of at least one target sorting action score corresponding to the at least one candidate video sub-segment is determined, and the average value is determined as the target sorting action score corresponding to the candidate video segment.

9. The device according to claim 6, wherein: The output unit is further configured to select a target video segment from the at least one candidate video segment based on at least one target sorting action score after the curve smoothing process in the following manner: Selecting, from the at least one candidate video segment, a candidate video segment whose corresponding target sorting action score after curve smoothing is greater than a preset score threshold; A preset second number of video segments are selected as target video segments from the candidate video segments having scores greater than the score threshold in descending order of the target sorting action scores after the corresponding curve smoothing processing.

10. The device according to any one of claims 6 to 9, wherein: The video information includes a time period corresponding to the video.

11. An electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

12. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images