Method and apparatus for determining text track in video, and device, and medium
By using the similarity of video history information and adjacent frame text bounding box in text detection in video, the problems of inaccurate identification and trajectory switching in the prior art are solved, and the accuracy and user experience of text trajectory recognition in video are improved.
Patent Information
- Application Number
- PCT/CN2024/092067
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2024-05-09
- Publication Date
- 2024-11-21
AI Technical Summary
The prior art has the problem of inaccurate identification of trajectories in text detection in videos, especially because text recognition and trajectory determination are coupled together, resulting in inaccurate text recognition and prone to trajectory switching.
By determining the feature set corresponding to the text set in the target frame and the text set in the previous frame, and calculating the correlation set, the text track is determined, the historical information in the video is used to improve the recognition accuracy, and the adjacent two frames of text are introduced. The similarity of bounding boxes to enhance prediction accuracy.
It improves the accuracy of text trajectory recognition in video, reduces text trajectory switching during the recognition process, improves user experience, and makes text recognition more comprehensive by separately processing text recognition and trajectory prediction.
Smart Images

Figure CN2024092067_21112024_PF_FP_ABST
Abstract
Description
Method, apparatus, device and medium for determining trajectory of text in video
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims priority to Chinese invention patent application number 202310563293.3, entitled “Methods, devices, apparatus and media for determining the trajectory of text in a video” and filed on May 18, 2023, and is incorporated herein by reference in its entirety. Technical Field
[0003] Embodiments of the present disclosure generally relate to the field of video processing, and more particularly to methods, devices, apparatuses, and media for determining the trajectory of text in a video. Background Art
[0004] With the rapid development of computer technology and the widespread use of video files, the detection of information in videos has become increasingly important. For example, detecting people and text in videos. To facilitate more in-depth analysis of videos, video text detection has become an important development direction in video detection. Video text detection mainly aims to locate and track text instances in videos. By locating and tracking text in videos, videos have greater application prospects in video analysis and multimedia information retrieval. Although great efforts have been made in both text detection and tracking, many problems still need to be solved because the detection of text in videos is affected by various factors.
[0005] Summary of the Invention
[0006] Embodiments of the present disclosure provide a method, apparatus, device, and medium for determining a trajectory of text in a video.
[0007] According to a first aspect of the present disclosure, a method for determining the trajectory of text in a video is provided. The method includes determining a first feature set corresponding to a first text set and a second feature set corresponding to a second text set, the first text set being located in a target frame of the video, the second text set being located in a plurality of frames preceding the target frame in the video, and the text in the second text set being associated with the text trajectory in the text trajectory set. The method also includes determining an association degree set based on the first feature set and the second feature set, the association degree in the association degree set indicating the likelihood that the text in the first text set is on the text trajectory in the text trajectory set. The method also includes determining a text trajectory corresponding to the text in the first text set based on the association degree set.
[0008] In a second aspect of the present disclosure, a device for determining a trajectory of text in a video is provided. The device includes a feature set determination module configured to determine a first feature set corresponding to a first text set and a second feature set corresponding to a second text set, wherein the first text set is located in a target frame of the video, the second text set is located in multiple frames before the target frame in the video, and the text in the second text set is associated with a text trajectory in a text trajectory set; an association degree set determination module configured to determine an association degree set based on the first feature set and the second feature set, wherein the association degree in the association degree set indicates the likelihood that the text in the first text set is on a text trajectory in the text trajectory set; and a target text trajectory determination module configured to determine a text trajectory corresponding to the text in the first text set based on the association degree set.
[0009] In a third aspect of the present disclosure, an electronic device is provided, comprising at least one processor; and a storage device for storing at least one program, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the method according to the first aspect of the present disclosure.
[0010] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.
[0011] It should be understood that the content described in this content section is not intended to limit the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.
[0013] FIG1 illustrates a schematic diagram of an example environment in which devices and / or methods according to embodiments of the present disclosure may be implemented;
[0014] FIG2 illustrates a schematic diagram of an example of detecting text trajectories according to an embodiment of the present disclosure;
[0015] FIG3 illustrates a schematic diagram of an example method for determining a trajectory of text in a video according to an embodiment of the present disclosure;
[0016] FIG4 illustrates a schematic diagram of an example framework for determining the trajectory of text in a video according to an embodiment of the present disclosure;
[0017] 5A-5C are schematic diagrams illustrating three examples of detecting similarity of text boxes according to an embodiment of the present disclosure;
[0018] FIG6 illustrates a schematic diagram of an example recognition result of a video text according to an embodiment of the present disclosure;
[0019] FIG7 illustrates a schematic diagram of an example recognition result of a video text according to an embodiment of the present disclosure;
[0020] FIG8 illustrates a schematic block diagram of an apparatus for determining a trajectory of text in a video according to an embodiment of the present disclosure;
[0021] FIG9 illustrates a schematic block diagram of an example device suitable for implementing embodiments of the present disclosure.
[0022] In the various drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION
[0023] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions. In response to receiving the user's active request, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require the acquisition and use of the user's personal information. Thus, the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium and other software or hardware that performs the operation of the technical solution of this disclosure based on the prompt message.
[0024] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0025] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0026] As described above, there are still many technical problems to be solved in the detection and tracking of text in video. To better identify text content in videos, conventional approaches have proposed a number of methods for identifying text in videos. For example, in some conventional approaches, the text in the video frame to be identified is compared with the text in an already identified video frame to determine whether the text in the video frame to be identified corresponds to the text in the already identified video frame. If a corresponding text is found, the trajectory of the text in the video frame to be processed is determined based on the trajectory of the text in the already identified video frame. In this process, the text content is also identified. However, this conventional approach only uses a comparison between two frames to determine the trajectory and content of the text. However, this approach suffers from the problem of inaccurate text trajectory. For example, text in the video to be processed may be associated with the wrong text in an already identified video frame, causing the text trajectory to switch. Furthermore, because this conventional approach couples text recognition and trajectory determination, this can also lead to inaccurate text recognition.
[0027] To at least address the above and other potential issues, embodiments of the present disclosure provide a method for determining text trajectories in a video. In this method, a computing device first determines a first feature set corresponding to a first text set and a second feature set corresponding to a second text set. The first text set is located in a target frame of the video, and the second text set is located in multiple frames preceding the target frame in the video. The text in the second text set is associated with text trajectories in a text trajectory set. The computing device also determines a set of associations based on the first and second feature sets. The associations in this set indicate the likelihood that the text in the first text set is located on a text trajectory in the text trajectory set. Using this set of associations, the computing device can determine the text trajectory corresponding to the text in the first text set. This method utilizes historical information in the video, improves the accuracy of text trajectory recognition, and reduces the occurrence of text trajectory switching during the recognition process. In a further improvement, the solution also incorporates the similarity between text bounding boxes in two adjacent frames, making text trajectory prediction more accurate. Furthermore, in this solution, text recognition and trajectory prediction are processed separately, making text recognition more comprehensive and improving the user experience.
[0028] Embodiments of the present disclosure will be described in further detail below with reference to the accompanying drawings. FIG. 1 illustrates an example environment in which devices and / or methods of embodiments of the present disclosure may be implemented. In environment 100 , a computing device 104 processes a video 102 to determine text tracks 114 for text in the video.
[0029] Examples of computing device 104 include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multi-processor systems, consumer electronics, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
[0030] As shown in FIG1 , video 102 includes multiple consecutive video frames, and there is recognizable text in the video frames. The number of video frames included in video 102 can be set by the user as needed. FIG1 shows that computing device 104 receives video 102, for example, from other computing devices or remote storage devices in the network, which is merely an example and not a specific limitation of the present disclosure. Computing device 104 can also obtain video 102 from a local storage device.
[0031] After receiving the video 102, the computing device 104 processes the video 102. During the processing, the computing device 104 determines text tracks formed by text in the video frames in the video 102. Additionally, the computing device 104 recognizes the text content in the video frames to generate corresponding text.
[0032] When processing the video 102, the computing device 104 will use the features of the text in the video frames that have been processed to form a global feature pool 106 for the video 102. The global feature pool 106 stores the features of the text in each of the processed video frames, such as the embedded vector representation of the text. The computing device 104 will also store the features of one or more texts in a video frame in an associated manner, as shown by row 108, which represents the storage of the features of one or more texts recognized in a video frame. In addition, after processing the video frame, the computing device 104 will also use the position of the same text in different frames in the video frame to generate text tracks, and store the text features with the same track in multiple frames in an associated manner, such as assigning a corresponding track identifier to each text feature, such as the features in box 110 are features with the same track. Therefore, there is a set of text tracks corresponding to the text features in the global feature pool in the computing device.
[0033] When processing the first frame of video 102, the computing device assigns a track identifier to each recognized text in the first frame. When processing subsequent frames, the computing device uses the first feature set of the text in the processed frame and a second feature set formed by features in the global feature pool 106 to determine a set of associations 112 indicating the likelihood that text in the first text set is on a text track in the text track set. After obtaining the set of associations 112, the computing device uses it to determine text tracks 114 corresponding to the text in the first text set.
[0034] By using this method, the historical information in the video is utilized, the accuracy of text trajectory recognition is improved, and the text trajectory switching that occurs during the recognition process is reduced, thereby improving the user experience.
[0035] 1 , a schematic diagram of an example environment in which the apparatus and / or method of an embodiment of the present disclosure may be implemented is described above. Below, a schematic diagram of an example of detecting text trajectories according to an embodiment of the present disclosure is described in conjunction with FIG. 2 .
[0036] As shown in FIG2 , in example 200, a computing device processes a video. For example, it processes video frames 202-1, 202-2, ..., 202-n in the video and generates a global feature pool 204. This feature pool 204 includes two text tracks 206 and 210. Each track has a corresponding text feature, such as feature 208. For the video frame 202-0 to be processed in the video, a text feature 212 is determined. By utilizing the method disclosed herein, it is determined that the text has a corresponding track 210.
[0037] A schematic diagram of an example of detecting text trajectories according to an embodiment of the present disclosure is described above in conjunction with FIG2 . A schematic diagram of an example method for determining text trajectories in a video according to an embodiment of the present disclosure is described below in conjunction with FIG3 . The process shown in FIG3 may be performed on the computing device 104 shown in FIG1 or any other suitable computing device.
[0038] At block 302, a first feature set corresponding to a first text set and a second feature set corresponding to a second text set are determined. The first text set is located in a target frame of a video, the second text set is located in a plurality of frames preceding the target frame in the video, and the text in the second text set is associated with text tracks in a set of text tracks. For example, the computing device 104 in FIG. 1 is configured to determine a first text feature set for a first text set in a target frame in the video 102 and a second feature set for a second text set in a plurality of video frames preceding the target frame.
[0039] In some embodiments, when determining the first feature set, the computing device 104 first determines the text area in the target frame. Then, the computing device 104 detects the image features corresponding to the target frame, and then determines the image features corresponding to the text area from the image features. Furthermore, the computing device 104 also uses the partial image features to determine the features corresponding to the text in the text area in the first text set. In one example, the computing device detects the text area and the text features in the text area through a set neural network model. In another example, the computing device 104 determines the text area through a mapping relationship between a predetermined text area and an image, and between image features and text features, and determines the image features and text features. The above examples are only used to describe the present disclosure, and are not specific limitations of the present disclosure. Those skilled in the art may use any appropriate method to determine.
[0040] In some embodiments, when determining the second feature set corresponding to the second text set, computing device 104 first determines the features of the text associated with each of the multiple frames preceding the target frame. Computing device 104 then generates the second feature set by concatenating the features of the text associated with each of the multiple frames. Alternatively or additionally, the features of the text associated with each of the multiple frames may be directly used as the second feature set. The above examples are merely illustrative of the present disclosure and are not intended to limit the present disclosure.
[0041] At block 304, a set of association degrees is determined based on the first feature set and the second feature set. A degree of association in the set of association degrees indicates a likelihood that text in the first text set is on a text track in the text track set. For example, the computing device 104 determines a set of association degrees indicating a likelihood that text in the target frame is on a text track in the text track set based on the first feature set in the target frame and the second feature set derived from features in the global feature pool 106.
[0042] In some embodiments, for ease of description, the association degree set is also referred to as the first association degree set. When determining the association degree set, the computing device first uses the first feature set and the second feature set to calculate the association degree between the text in the first text set and the text in the second text set to generate a second association degree set. The association degree in the second association degree set indicates the similarity between the text in the first text set and the text in the second text set. The computing device 104 then determines the first association degree set based on the second association degree set and the text trajectory set. In one example, when determining the second association degree set, the computing device determines a third feature set based on the association relationship between the text in the second feature set and the second text set. For example, the computing device inputs the second feature set into the encoder of the Transformer network, and the encoder generates a third feature set based on the association relationship between the text in the text set. The computing device then determines the second association degree set based on the third feature set and the first feature set. For example, the third feature set and the first feature set are input into the decoder of the Transformer network to obtain the second association degree set.
[0043] In some embodiments, the computing device 104 first obtains a predetermined mapping relationship between a predetermined feature set and a degree of association, and then searches for a second degree of association set corresponding to the first feature set and the second feature set according to the mapping relationship.
[0044] In some embodiments, when determining the first degree of association set, the computing device first selects a first degree of association subset corresponding to the target text in the first text set from the second degree of association set. For example, all degrees of association corresponding to the target text are found from the second degree of association set to form a first degree of association subset. Then, the computing device 104 selects a second degree of association subset corresponding to the target text track in the text track set from the first degree of association subset. For example, the computing device 104 then searches the first degree of association subset for the degree of association related to the target track based on the identifier of the target track. Then, the computing device uses the second degree of association subset to determine the degree of association between the target text and the target text track.
[0045] In some embodiments, the associations in the second association subset are probability values. Therefore, when determining the association between the target text and the target text trajectory, the computing device will determine the association between the target text and the target text trajectory by adding the probability values in the second association subset.
[0046] In some embodiments, the associations in the second association subset are probability values or association levels. When determining the association between the target text and the target text track, the computing device selects the largest value in the second association subset as the association between the target text and the target text track. The above examples are merely for describing the present disclosure and are not intended to limit the present disclosure.
[0047] In some embodiments, when determining the second set of associations based on the first feature set and the second feature set, the computing device determines the second set of associations by applying a neural network model to the first text set and the second text set, where the neural network model includes an encoder and a decoder, such as a Transformer neural network model.
[0048] Furthermore, the computing device 104 can also train the neural network model. For example, the computing device first determines a set of sample features corresponding to a set of text in multiple frames of a sample video, as well as a set of sample correlations between the sample feature set and the neural network model. The computing device then uses the sample feature set as sample inputs to the encoder and decoder, respectively, and uses the sample correlation set as the sample output of the neural network model to train the neural network model.
[0049] In some embodiments, the computing device 104 first obtains a mapping relationship between a predetermined feature set and a correlation degree, and then searches for a first correlation degree set corresponding to the first feature set and the second feature set based on the mapping relationship. The above examples are only used to describe the present disclosure and are not intended to limit the present disclosure.
[0050] At block 306 , text tracks corresponding to text in the first text set are determined based on the set of associations. For example, the computing device 104 determines text tracks corresponding to text in the first text set based on the set of associations 112 .
[0051] In some embodiments, computing device 104 directly uses the association set to determine text tracks corresponding to text in the first text set. In one example, the Hungarian algorithm is applied to the association set to match text in the first text set with text tracks in the text track set. In another example, a mapping relationship between text and text tracks, pre-set based on the association values between the text and text tracks, is first obtained, and then the text track corresponding to the text in the first text set is searched within the mapping relationship based on the association set.
[0052] In some embodiments, when determining a text track corresponding to text in a first text set, the computing device first determines a similarity between a first bounding box of the first text and a second bounding box of a second text, where the first text is included in the first text set and the second text is included in the second text set and is located in a frame previous to the target frame. The computing device then determines a text track corresponding to the text in the first text set based on the association set and the similarity.
[0053] In some embodiments, when determining the similarity between a first bounding box of a first text and a second bounding box of a second text, the computing device first determines a first representation indicating the size and position of the first bounding box and a second representation indicating the size and position of the second bounding box. For example, the size and position of the bounding box may be represented as b(x, y, w, h, θ), where x and y represent the coordinates of the center point of the bounding box, w represents one side of the bounding box, h represents another side of the bounding box, and θ represents the rotation angle of the bounding box about the x-axis. The computing device may then use the first and second representations to determine the similarity between the first bounding box and the second bounding box.
[0054] In some embodiments, when determining the similarity between the first bounding box and the second bounding box, the computing device converts the first representation into a first Gaussian distribution and then converts the second representation into a second Gaussian distribution. The computing device then determines the similarity based on the first Gaussian distribution and the second Gaussian distribution, for example, by calculating a distance between the two Gaussian distributions, where the distance indicates the similarity between the two, such as the Wasserstein distance.
[0055] In some embodiments, when determining the similarity between the first bounding box and the second bounding box, the computing device searches for the similarity corresponding to the first representation and the second representation in a preset mapping relationship between the representation of the bounding box and the similarity.
[0056] In some embodiments, when using the association degree set and the similarity to determine the text track corresponding to the text in the first text set, the computing device first determines the first association degree between the text track where the second text is located and the first text from the association degree set. For example, the computing device searches the association degree set for the first association degree corresponding to the text track where the second text is located and the first text. Then, the computing device determines the second association degree between the text track where the second text is located and the first text based on the similarity and the first association degree. Finally, the second association degree is used to update the first association degree in the association degree set. For example, the computing device determines the maximum value of the similarity and the second association degree as the association degree between the text track and the first text. The computing device uses the updated association degree set to determine the text track corresponding to the text in the first text set.
[0057] In some embodiments, when determining text tracks corresponding to text in the first text set, the computing device may determine the text tracks corresponding to the text in the first text set by performing a maximum match between the first text set and the text track set based on the updated association degree set. For example, the text tracks corresponding to the text in the first text set may be determined using a Hungarian algorithm.
[0058] In some embodiments, the computing device performs video understanding operations on the video based on the text track. In some embodiments, the computing device performs video retrieval based on the text track. In some embodiments, the computing device performs video capture based on the text track. The above examples are merely illustrative of the present disclosure and are not intended to limit the present disclosure.
[0059] This method, by leveraging historical information in the video, improves the accuracy of text trajectory recognition and reduces the occurrence of text trajectory switching during the recognition process. In a further improvement, the solution also incorporates the similarity of text bounding boxes between adjacent frames, making text trajectory prediction even more accurate. Furthermore, this solution handles text recognition and trajectory prediction separately, making text recognition more comprehensive and improving the user experience.
[0060] The above describes a schematic diagram of an example method for determining the trajectory of text in a video according to an embodiment of the present disclosure in conjunction with Figure 3. The following describes a schematic diagram of an example framework for determining the trajectory of text in a video according to an embodiment of the present disclosure in conjunction with Figure 4.
[0061] As shown in FIG4 , example 400 employs an end-to-end framework for video text recognition, which performs global information association and utilizes shape-aware information. The entire framework is shown in FIG4 . The framework includes three parallel heads: a detection head, a recognition head, and a tracking head. Given a video, including video frames 402-0, 402-1, ..., 402-n-1, and 402-n, also referred to as frame F t , F t-1 ,…F t-n-1 , F t-n , where n is a positive integer, and the n+1 video frames can be the complete video or the video frames included in the sliding video window in the video. First, the backbone network 404 and the detection head 406 are used to process each video frame in the video to detect the bounding box corresponding to the text. As shown in FIG4 , the processed video frames 408-0, 408-1, ..., 408-n-1 and 408-n include the bounding box for the text. In frame 402-0 or frame F t When the detected bounding boxes include potential text objects, extract corresponding text features or track embeddings for each object The tracking embedding in each frame forms a feature set 410-0, 410-1, ..., 410-n-1 and 410-n. In addition, a global embedding pool 412 is set as Used to store the features or track embeddings of text in processed video frames. The concatenation of all embeddings in is represented as in represent The number of embeddings in N.t It is frame F t is the number of texts in , d is the dimension of each embedding, and L is the sliding window size.
[0062] The tracking head in this frame is used to calculate the distance between the object in the frame Ft and The association scores between the objects in generate the association matrix Then P Asso Converted to the expression The trajectory-level correlation matrix, where K t is the global embedding pool In addition, it is necessary to calculate the number of trajectories in two adjacent frames F t and F t-1 The morphological similarity between 428 is calculated by calculating the Wasserstein distance 426 to output the distance matrix Two score matrices P Tracklet and W t The final score is generated by a simple maximum operation 424, without any other post-processing, and outputting it as shown in Figure 4. And apply a maximum matching algorithm, such as the Hungarian algorithm, to t Assign text track identifiers to text objects in .
[0063] For the global embedding pool The text feature set in and the text feature set 410-0 are processed using a neural network model to determine the relevance between the texts corresponding to the two feature sets. For example, a Transformer network 416 is used for global tracking. All previous embeddings in the global embedding pool are encoded as historical information or intermediate representations by the Transformer encoder 416. The Transformer decoder 420 inputs the encoded historical information and uses the embedding in the current frame as a query to calculate the similarity score between the text in the current frame and the trajectory. Furthermore, a multiplication operation 422 is also performed when calculating the similarity score. The entire process is shown in formulas (1) and (2): H t =Encoder(G t ) (1)
[0064] in Indicates the historical information of the encoding. is the output incidence matrix. refers to the query embedding (i.e., the embedding of the current frame).
[0065] During training, all embeddings are associated with themselves (i.e., Qt =G t ), generate the correlation matrix in is the number of all texts in a batch of frames, B represents the number of frames in a batch of frames, for example, this number can be 16. For each text in each timestamp t, set the vector It represents a query embedding q i and frame F t The extra dimension in a refers to the empty association (i.e., when there is no matching target in the query frame, which usually indicates occlusion or end of track). In formula (3), the Softmax function is used to convert the score into a probability P Asso :
[0066] Therefore, the score is learned by minimizing the log-likelihood of each possibility. For each trajectory set k=1,2,...,K,assuming T k There are N k embeddings, for the global embedding pool Each query embedding in e i , only in e i ∈T k The loss is calculated when . The loss is expressed by formula (4):
[0067] Where if the query embedding belongs to the trajectory set is 1, otherwise it is 0.
[0068] Furthermore, to increase the discriminative power of embeddings, semantic information is also introduced to improve tracking performance. Specifically, the embedding of the input Transformer is extracted by rotating RoIAlign with a convolutional layer and a long short-term memory (LSTM) layer, followed by a fully connected layer to project the features into word categories. The architecture is represented by formula (5): t =lstm(conv(r-roi(X t ))) o t =fc(e t ) (5)
[0069] where X t It is frame F t The backbone feature map in e t is embedded into Transformer for association, and o t is the recognition output supervised by the Connectionist Temporal Classification (CTC) loss.
[0070] The query in this scheme carries information about each text instance, and attention is performed between the query and the global embedding to output a correlation matrix, thereby achieving a stronger perception of contextual information. This scheme also maintains a global embedding pool, which can be viewed as a queue and has long-range temporal awareness. This scheme also avoids the competition between the two tasks of determining the trajectory and recognizing the text, which can harm detection performance.
[0071] Regarding morphological information, the Wasserstein distance is used to model positional and shape similarity. As shown in Figures 5A-5C , which depict three examples of detecting text box similarity according to embodiments of the present disclosure, the Wasserstein distance is used to demonstrate the recognition capabilities of the text box. In Figure 5A , all text moves at a low speed, so positional information is sufficient for association. For example, the position and shape of the text in video frames 502 and 504 remain essentially unchanged, with the position and shape changes shown in box 506. In this case, the Wasserstein distance can correctly match the text, as shown in box 508. In Figure 5B , the rapid movement of the camera causes the text to drift rapidly. In this case, while the morphological differences are obvious, the Wasserstein distance can capture the inconsistencies. For example, the position and shape of the text in video frames 510 and 512 vary significantly, with the position and shape changes shown in box 514. In this case, the Wasserstein distance can correctly match the text, as shown in box 516. In Figure 5C , despite the rapid movement of the car, the Wasserstein distance can still correctly perform positional association. For example, the position and shape of the text in video frames 518 and 520 have greater changes, as shown in box 522. At this point, the Wasserstein distance can still be correctly matched, as shown in box 524.
[0072] In order to obtain the perception of position and shape as described above, it is necessary to model the polygons in different frames as Gaussian distributions and measure the similarity by distribution distance. For each bounding box in two adjacent frames, calculate the pairwise Wasserstein distance between the corresponding bounding boxes b(x, y, w, h, θ). Convert the rotated box b to a Gaussian distribution as shown in formula (6)
[0073] Where R is the rotation matrix and S is the diagonal matrix. The Wasserstein distance between two Gaussian distributions is expressed by formula (7):
[0074] Where T rRepresents the matrix. In order to utilize appropriate angles and coordinates, Wasserstein distance can capture position similarity and shape similarity. Finally, in order to convert the distance into an applicable position score, formula (8) is used:
[0075] Where α is a hyperparameter and f is a distance normalization function. Set f(σ1,σ2)=(T r (σ1σ2)) 1 / 4 .
[0076] Returning to Figure 4 to continue the description, the loss function is also calculated during the training process. This framework has three tasks and three corresponding losses. For the detection head, L1 loss is used. For other losses, they are set as needed. For the recognition head, Connectionist Temporal Classification (CTC) text loss is used. The loss of the entire framework is shown in formula (9):
[0077] Among them, σ1, σ2, σ3 are learnable parameters, is the regularization term of the noise term.
[0078] During inference, the trajectory is constructed iteratively. For the initial frame F0, each text is considered as the beginning of the trajectory. t , set up a global embedding pool It stores all previous embeddings in a sliding window, and the corresponding embedding matrix is written as G t (i.e., connected embedding). Assume There is K t trajectories, each with N k Embed the current frame F t The embedding matrix is considered as Q t In Q t and G t Calculate the association score A between t , and converted into a probability matrix P through the Softmax function Asso Then the trajectory-level sum is applied to calculate the probability of each trajectory. Calculate using formula (10):
[0079] where e j is the embedding in each trajectory. After embedding all the vectors in , we can finally get a matrix represents the probability that the embedding vector belongs to the trajectory. In addition, the frame F t-1 and F tThe Wasserstein distance between them is calculated as the position and shape similarity score W t , the final output is max(P Tracklet ,W t ). The Hungarian algorithm is applied to ensure that the ID assignment is unique for each text.
[0080] FIG6 illustrates a schematic diagram of an example recognition result of video text according to an embodiment of the present disclosure. In example 600, in four video frames 602, 604, 606, and 608 of a video recognized by the solution of the present disclosure, the track of each text in the four video frames is correctly identified without track switching errors. Different colored bounding boxes are used in the figure to represent different text tracks. As shown in boxes 610, 612, 614, and 616, the text within these boxes corresponds to the same text track.
[0081] FIG7 illustrates a schematic diagram of an example recognition result of video text according to an embodiment of the present disclosure. In example 700, two video frames 702 and 704 in a video are processed. Because text recognition and text track confirmation are performed separately, as shown in the two frames, the text in the image is correctly identified.
[0082] FIG8 shows a schematic block diagram of an apparatus for determining the trajectory of text in a video according to an embodiment of the present disclosure. As shown in FIG8 , the apparatus 800 includes a feature set determination module 802 configured to determine a first feature set corresponding to a first text set and a second feature set corresponding to a second text set, wherein the first text set is located in a target frame of the video, the second text set is located in a plurality of frames before the target frame in the video, and the text in the second text set is associated with the text trajectory in the text trajectory set; an association degree set determination module 804 configured to determine an association degree set based on the first feature set and the second feature set, wherein the association degree in the association degree set indicates the possibility that the text in the first text set is on the text trajectory in the text trajectory set; and a text trajectory determination module 806 configured to determine the text trajectory corresponding to the text in the first text set based on the association degree set.
[0083] In some embodiments, the association degree set is a first association degree set, and the association degree set determination module 804 includes: a second association degree set determination module, configured to determine the second association degree set based on the first feature set and the second feature set, the association in the second association degree set indicating the similarity between the text in the first text set and the text in the second text set; and a first association degree set determination module, configured to determine the first association degree set based on the second association degree set and the text trajectory set.
[0084] In some embodiments, the second association set determination module includes: a third feature set determination module, configured to determine the third feature set based on the association relationship between the second feature set and the text in the second text set; and a feature-based association set determination module, configured to determine the second association set based on the third feature set and the first feature set.
[0085] In some embodiments, the first association degree set determination module includes: a first association degree subset determination module, configured to select a first association degree subset corresponding to the target text in the first text set from the second association degree set; a second association degree subset determination module, configured to select a second association degree subset corresponding to the target text trajectory in the text trajectory set from the first association degree subset; and a text and trajectory association degree determination module, configured to determine the association degree between the target text and the target text trajectory based on the second association degree subset.
[0086] In some embodiments, the associations in the second association subset are probability values, and the text-trajectory association determination module includes: an addition module configured to determine the association between the target text and the target text trajectory by adding the probability values in the second association subset.
[0087] In some embodiments, the second association set determination module includes: a neural network model application module, configured to determine the second association set by applying a neural network model to the first text set and the second text set, the neural network model including an encoder and a decoder.
[0088] In some embodiments, the device 800 also includes: a sample determination module, configured to determine a sample feature set corresponding to a sample text set in multiple sample frames in a sample video and a sample association set between the sample feature set and itself; and a training module, configured to train the neural network model by using the sample feature set as the sample input of the encoder and the decoder respectively and using the sample association set as the sample output of the neural network model.
[0089] In some embodiments, the text trajectory determination module includes: a first similarity determination module, configured to determine the similarity between a first bounding box of a first text and a second bounding box of a second text, the first text being included in a first text set, the second text being included in a second text set and being located in a previous frame of a target frame; and a first text trajectory determination module, configured to determine a text trajectory corresponding to the text in the first text set based on the association set and the similarity.
[0090] In some embodiments, the first similarity determination module includes: a first representation determination module configured to determine a first representation for indicating the size and position of the first bounding box; a second representation determination module configured to determine a second representation for indicating the size and position of the second bounding box; and a second similarity determination module configured to determine the similarity between the first bounding box and the second bounding box based on the first representation and the second representation.
[0091] In some embodiments, the second similarity determination module includes: a first Gaussian distribution determination module configured to convert the first representation into a first Gaussian distribution; a second Gaussian distribution determination module configured to convert the second representation into a second Gaussian distribution; and a third similarity determination module configured to determine similarity based on the first Gaussian distribution and the second Gaussian distribution.
[0092] In some embodiments, the first text trajectory determination module includes: a first correlation determination module, configured to determine the first correlation between the text trajectory where the second text is located and the first text from the correlation set; a second correlation determination module, configured to determine the second correlation between the text trajectory where the second text is located and the first text based on the similarity and the first correlation; an update module, configured to use the second correlation to update the first correlation in the correlation set; and an updated text trajectory determination module, configured to determine the text trajectory corresponding to the text in the first text set based on the updated correlation set.
[0093] In some embodiments, the updated text trajectory determination module includes: a maximum matching determination module configured to determine the text trajectory corresponding to the text in the first text set by performing maximum matching on the first text set and the text trajectory set based on the updated association set.
[0094] In some embodiments, the feature set determination module 802 includes: a text area determination module, configured to determine the text area in the target frame; an image feature determination module, configured to determine image features corresponding to the text area; and a text feature determination module, configured to determine features corresponding to the text in the text area in the first text set based on the image features.
[0095] In some embodiments, the feature set determination module 802 includes: a multi-frame text feature determination module, configured to determine features of text associated with each frame in a plurality of frames; and a second feature set generation module, configured to generate a second feature set by connecting features of text associated with each frame in a plurality of frames.
[0096] In some embodiments, the apparatus 800 further includes: a video operation unit configured to perform at least one of the following on the video based on the text track: video understanding, video retrieval, or video capture.
[0097] FIG9 shows a schematic block diagram of an example device 900 that can be used to implement an embodiment of the present disclosure. The computing device 104 in FIG1 can be implemented using device 900. As shown in the figure, device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 902 or loaded from a storage unit 908 into a random access memory (RAM) 903. In RAM 903, various programs and data required for the operation of device 900 can also be stored. CPU 901, ROM 902, and RAM 903 are connected to each other via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 904.
[0098] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.
[0099] The various processes and processing described above, such as method 300, may be performed by processing unit 901. For example, in some embodiments, method 300 may be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of method 300 described above may be performed.
[0100] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.
[0101] Computer-readable storage media can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. The computer-readable storage media used herein is not to be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by a fiber optic cable), or an electrical signal transmitted by a wire.
[0102] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.
[0103] The computer program instructions for performing the disclosed operation can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as, for example, utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present disclosure.
[0104] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0105] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0106] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0107] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0108] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A method for determining the trajectory of text in a video, include: Determining a first feature set corresponding to a first text set and a second feature set corresponding to a second text set, wherein the first text set is located in a target frame of the video, the second text set is located in a plurality of frames before the target frame in the video, and texts in the second text set are associated with text tracks in a text track set; Determining a set of association degrees based on the first feature set and the second feature set, wherein the association degrees in the set of association degrees indicate the likelihood that text in the first text set is on a text track in the text track set; and Based on the association degree set, a text track corresponding to the text in the first text set is determined.
2. The method according to claim 1, wherein the association degree set is a first association degree set, wherein determining the association degree set include: Determining a second set of association degrees based on the first feature set and the second feature set, wherein the association degrees in the second set of association degrees indicate similarities between texts in the first text set and texts in the second text set; The first association degree set is determined based on the second association degree set and the text trajectory set.
3. The method according to claim 2, wherein determining a second set of association degrees include: Determine a third feature set based on the association relationship between the second feature set and the texts in the second text set; as well as The second set of association degrees is determined based on the third set of features and the first set of features.
4. The method according to claim 2, wherein determining the first set of association degrees include: Selecting a first relevance subset corresponding to the target text in the first text set from the second relevance set; Select a target text in the text trajectory set from the first association degree subset A second correlation degree subset corresponding to the trajectory; as well as Based on the second association degree subset, the association degree between the target text and the target text trajectory is determined.
5. The method according to claim 4, wherein the relevance in the second relevance subset is a probability value, wherein the relevance between the target text and the target text trajectory is determined include: The degree of association between the target text and the target text trajectory is determined by adding the probability values in the second degree of association subset.
6. The method according to claim 2, wherein a second set of associations is determined based on the first set of features and the second set of features. include: The second set of associations is determined by applying a neural network model to the first text set and the second text set, the neural network model comprising an encoder and a decoder.
7. The method according to claim 6, further comprising: include: Determine a sample feature set corresponding to a sample text set in a plurality of sample frames in a sample video and a sample correlation degree set between the sample feature set and itself; as well as The neural network model is trained by using the sample feature sets as sample inputs of the encoder and the decoder respectively and using the sample correlation degree sets as sample outputs of the neural network model.
8. The method of claim 1, wherein determining a text track corresponding to text in the first text set comprises: Determining a similarity between a first bounding box of a first text included in the first text set and a second bounding box of a second text included in the second text set and located in a previous frame of the target frame; and Based on the association degree set and the similarity, a text track corresponding to the text in the first text set is determined.
9. The method of claim 8, wherein determining the similarity between a first bounding box of the first text and a second bounding box of the second text include: determining a first representation indicating a size and a position of the first bounding box; determining a second representation indicating a size and a position of the second bounding box; as well as Determine the first border and the second border based on the first representation and the second representation The similarity between bounding boxes.
10. The method of claim 9, wherein determining the similarity between the first bounding box and the second bounding box include: converting the first representation into a first Gaussian distribution; converting the second representation into a second Gaussian distribution; as well as The similarity is determined based on the first Gaussian distribution and the second Gaussian distribution.
11. The method according to claim 8, wherein based on the association degree set and the similarity, a text track corresponding to the text in the first text set is determined include: Determining a first correlation degree between the text track where the second text is located and the first text from the correlation degree set; Determine a second degree of association between the text track where the second text is located and the first text based on the similarity and the first degree of association; Using the second degree of association to update the first degree of association in the set of degrees of association; as well as A text track corresponding to text in the first text set is determined based on the updated set of association degrees.
12. The method according to claim 11, wherein a text track corresponding to the text in the first text set is determined based on the updated set of associations. include: Based on the updated association degree set, a text track corresponding to the text in the first text set is determined by performing maximum matching on the first text set and the text track set.
13. The method according to claim 1, wherein determining a first feature set corresponding to a first text set include: Determining a text area in the target frame; determining image features corresponding to the text region; as well as Features corresponding to text in the text region in the first text set are determined based on the image features.
14. The method according to claim 1, wherein determining a second feature set corresponding to a second text set include: determining characteristics of text associated with each of the plurality of frames; as well as The second set of features is generated by concatenating features of text associated with each of the plurality of frames.
15. The method according to claim 1, further comprising: include: Based on the text track, at least one of the following is performed on the video: video understanding, video retrieval, or video capture.
16. A device for determining the trajectory of text in a video, include: a feature set determination module, configured to determine a first feature set corresponding to a first text set and a second feature set corresponding to a second text set, wherein the first text set is located in a target frame of the video, the second text set is located in a plurality of frames before the target frame in the video, and texts in the second text set are associated with text tracks in a text track set; A relevance set determination module is configured to determine a relevance set based on the first feature set and the second feature set, wherein the relevance in the relevance set indicates the possibility that the text in the first text set is on the text track in the text track set; as well as The target text track determination module is configured to determine a text track corresponding to the text in the first text set based on the association degree set.
17. An electronic device, include: at least one processor; as well as A storage device, used to store at least one program, when the at least one program is executed by the at least one processor, so that the at least one processor implements the method according to any one of claims 1-15.
18. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 15 when executed by a processor.
Citation Information
Patent Citations
Video character target tracking method based on online enhancement detection and electronic equipment
CN111931571A
Video text tracking method and device
CN112101344A
Video character tracking method and device, video processing method and device, equipment and medium
CN113392689A
Video character tracking method and device, electronic equipment and storage medium
CN114463376A
Method and device for determining track of text in video, equipment and medium
CN116567301A
