Method, apparatus, device and storage medium for video processing

Abstract

Embodiments of this disclosure provide a method, apparatus, device, and storage medium for video processing. The method includes: acquiring a video, the video including a plurality of first objects and a plurality of second objects; extracting a plurality of video segments from the video based on the interaction relationships between the plurality of first objects and the plurality of second objects, wherein in each of the plurality of video segments, at least one first object interacts with one second object; and determining, based on the plurality of video segments, the positional information of each of the plurality of first objects and the plurality of second objects in each frame of the video. In this manner, the accuracy of object segmentation in the video can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to methods, apparatus, devices, and storage media for video processing. Background Technology

[0002] With the development of computer vision and artificial intelligence technologies, various video-based processing methods have become an important component of intelligent systems. By processing and analyzing video content, information related to the scene environment, human actions, and object states can be extracted from continuous visual information, thereby supporting a variety of intelligent applications. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for video processing is provided. The method includes: acquiring a video, the video including a plurality of first objects and a plurality of second objects; extracting a plurality of video segments from the video based on the interaction relationships between the plurality of first objects and the plurality of second objects, wherein in each of the plurality of video segments, at least one first object interacts with a second object; and determining, based on the plurality of video segments, position information of each of the plurality of first objects and the plurality of second objects in each frame of the video.

[0004] In a second aspect of this disclosure, an apparatus for video processing is provided. The apparatus includes: an acquisition module configured to acquire a video, the video including a plurality of first objects and a plurality of second objects; an extraction module configured to extract a plurality of video segments from the video based on interaction relationships between the plurality of first objects and the plurality of second objects, wherein in each of the plurality of video segments, at least one first object interacts with one second object; and a determination module configured to determine, based on the plurality of video segments, the position information of each of the plurality of first objects and the plurality of second objects in each frame of the video.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method of the first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program that can be executed by a processor to perform the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this summary section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown; Figure 2 A schematic diagram of a process for video processing according to some embodiments of the present disclosure is shown; Figure 3 A schematic diagram illustrating a process for determining location information according to some embodiments of the present disclosure is shown; Figure 4 A flowchart of a process for video processing according to some embodiments of the present disclosure is shown; Figure 5 A block diagram of an apparatus for video processing according to some embodiments of the present disclosure is shown; and Figure 6 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0010] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0011] It should be noted that the acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0012] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0013] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably.

[0014] Generally, machine learning can be divided into three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0015] The following description will focus on exemplary embodiments of the present disclosure with reference to the accompanying drawings.

[0016] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, server 130 is capable of establishing a communication connection with one or more terminal devices (collectively or individually referred to as electronic devices 110). At least one application 120 may be installed on electronic device 110. User 140 may interact with application 120 via electronic device 110 and / or attached devices of electronic device 110. Application 120 may be a content presentation application, an online shopping application, or any other suitable application.

[0017] exist Figure 1In environment 100, application 120 can also be accessed in other ways, such as through a webpage. If application 120 is active, electronic device 110 can display the interface 150 of application 120. Interface 150 may include various interfaces that application 120 can provide, such as content presentation interface, content creation interface, service request submission interface, request processing result interface, message interface, personal homepage, etc.

[0018] In some embodiments, electronic device 110 communicates with server 130 to provide services to application 120. Electronic device 110 can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, electronic device 110 can also support any type of user-facing interface (such as "wearable" circuitry). Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0019] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0020] As briefly mentioned above, with the development of computer vision technology, video data processing and analysis have been widely applied in various scenarios. For example, in human-computer interaction, behavior understanding, and intelligent analysis, it is often necessary to identify and determine the positional relationships of different objects in each frame of a video to support subsequent understanding and decision-making. In these application scenarios, videos often contain multiple objects, and there may be interactive relationships between different objects. By analyzing the objects and their interactive relationships in the video, and further determining the positional information of each object in the video, it is helpful to improve the accuracy of video content understanding and processing effectiveness.

[0021] In related technologies, a typical implementation involves frame-by-frame processing based on models and large-scale labeled data. This method determines the positional information of objects in each frame by performing object detection or segmentation on each frame of the video. Another typical implementation involves combining object detection with temporal processing techniques to track target objects in the video and propagate the initial detection results over time to obtain the changes in object position across frames.

[0022] In the aforementioned situations, objects throughout the entire video are typically processed uniformly. This approach struggles to distinguish object relationships across different interaction phases, easily leading to information interference between phases and affecting the accuracy of object location information. For example, hand-object interactions inherently possess phased and object-specific characteristics; a single video often contains multiple independent operational units involving interactions between different objects at different times. For instance, the right hand opens object A, and then the right hand retrieves object B from A. Current video processing techniques struggle to guarantee the consistency and continuity of the segmentation of each object's interaction cycle within a long video.

[0023] Furthermore, when performing cross-frame processing, related techniques typically propagate information over time based on the initial information of a single frame, lacking explicit modeling of this temporal information. When processing long videos containing multiple interaction phases, it is difficult to guarantee the temporal consistency of segmentation results within the same interaction phase. For example, in the process of "opening object A with the right hand," it is expected that object A will be continuously identified and segmented throughout the entire phase from the hand gradually approaching object A to the completion of the operation. However, in the phase before the hand makes contact with object A, processing methods based on single-frame information may fail to reliably identify the object, leading to discontinuities or instability in the segmentation results over time.

[0024] Furthermore, in determining object location information, related techniques typically rely on relatively coarse object region information (e.g., bounding boxes) as initial cues. Such cues fail to accurately reflect the actual shape characteristics of the object, thus affecting the accuracy of subsequent processing results.

[0025] In view of this, in order to at least partially solve the above problems, this disclosure provides a solution for video processing. According to various embodiments of this disclosure, a video to be processed can be obtained, the video including a plurality of first objects and a plurality of second objects. Further, based on the interaction relationships between the plurality of first objects and the plurality of second objects, a plurality of video segments are extracted from the video, in each of the plurality of video segments, at least one first object interacts with one second object. Further still, based on the plurality of video segments, the position information of each of the plurality of first objects and the plurality of second objects in each frame of the video is determined.

[0026] In this way, by dividing the video based on the interaction relationship between the first and second objects, different interaction processes in the video can be processed in relatively independent video segments. This helps reduce the mutual influence between different interaction processes and improves the accuracy of object location information determination. Furthermore, in the process of determining the location information of each object in each frame based on the video segments, different video segments can be processed separately, and the processing results can be integrated by combining the temporal relationship between the video segments. This helps improve the stability of cross-frame processing and the consistency of results. Moreover, through the above method, the adaptability to complex video scenes can be improved while ensuring processing effectiveness.

[0027] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0028] Figure 2 A flowchart of a process 200 for extracting video segments according to some embodiments of the present disclosure is shown. Figure 2 In this context, process 200 can be implemented at electronic device 110. For ease of description, the following explanation will use the execution of process 200 by electronic device 110 as an example. The following will combine... Figure 1 This disclosure provides a detailed description of the video processing solutions provided.

[0029] In some embodiments, the electronic device 110 may acquire video 202 to be processed. Video 202 may be an image sequence comprising multiple consecutive frames, such as video data captured by a camera or video data read from a storage device or server. In some examples, video 202 may be a first-person or third-person video and may include multiple first objects and multiple second objects. In video 202, the multiple first objects and multiple second objects may appear in different frames and may undergo positional or state changes during video playback. Interaction may occur between the first objects and the second objects in video 202. For example, the interaction may manifest as the first object touching, grasping, pushing, manipulating, or changing the spatial relationship of the second object.

[0030] In some embodiments, the first object can be an object performing an operation. For example, the first object can be a hand. Multiple first objects can include a left hand and a right hand. The second object can be an object in the video that interacts with the hand, such as an object being grasped, moved, or manipulated. Multiple second objects can include left-hand interactive objects, right-hand interactive objects, or two-hand interactive objects, wherein a left-hand interactive object represents an object interacting with the left hand, a right-hand interactive object represents an object interacting with the right hand, and a two-hand interactive object represents an object interacting with both the left and right hands simultaneously.

[0031] In the above scenario, multiple objects may exist in the video, and these objects have different positional relationships at different stages. If all objects are processed uniformly across the entire video, the accuracy and stability of object location information determination may be reduced. If the video can be reasonably divided so that different interaction stages are processed within relatively independent scopes, it can help reduce the mutual influence between different interaction stages, thereby improving the accuracy and stability of object location information determination. Therefore, in some embodiments, the electronic device 110 can segment the video before subsequent processing.

[0032] In some embodiments, the electronic device 110 may first detect and track objects in the video to obtain a first recognition result 204. For example, the corresponding object identifiers and bounding boxes of multiple first objects and multiple second objects in each frame of the video may be determined. By determining the first recognition result 204, the occurrence and motion trajectory of each object in consecutive frames may be obtained.

[0033] As an example, the first recognition result 204 can be obtained jointly through object detection and multi-object tracking processing. Exemplarily, the electronic device 110 can first detect objects in each frame of the video 202 to determine the object's location and category information. Further, the electronic device 110 can track the detected objects through cross-frame association to determine the correspondence of the same object between different frames. Through the above processing, a structured result containing detection and tracking information can be obtained. For example, the first recognition result 204 may include: object category (e.g., left hand, right hand, left-hand interaction object, right-hand interaction object, or two-handed interaction object), two-dimensional bounding box, and a unique object identifier. The object identifier can be used to associate the same object between different frames to characterize the continuous change process of the object in the video.

[0034] In some embodiments, the electronic device 110 can determine the interaction relationship between objects based on corresponding object identifiers and bounding boxes. The electronic device 110 can determine the spatial positional relationship, contact relationship, or distance change between the first object and the second object, and further determine whether the two interact. For example, if the bounding box of the first object and the bounding box of the second object are close in space, overlap, or meet a predetermined distance condition, it can be determined that there is an interaction relationship between the first object and the second object.

[0035] Furthermore, the electronic device 110 can extract multiple video segments 210 from the video based on the interaction relationships between multiple first objects and multiple second objects. For example, if a change is detected in the second object interacting with a certain first object, the corresponding time point can be used as a video segmentation point, thereby dividing the video into multiple video segments. In the extracted multiple video segments, each video segment can correspond to a different interaction stage, and in each video segment, at least one first object interacts with one second object. In this way, different interaction stages can be processed in relatively independent video segments, thereby providing a more stable basis for determining the subsequent object location information.

[0036] In some embodiments, the electronic device 110 can traverse each frame of the video and detect whether the second object interacting with the first object has switched. For example, for each frame of the video, the object identity of the second object currently interacting with each first object in that frame can be determined. If multiple first objects are detected interacting with the same second object simultaneously in a frame, the second object can be identified as the object commonly interacted with by the multiple first objects, and the object identity of the second object can be associated with the corresponding first object. If the object identity of the second object interacting with by a first object in the current frame is inconsistent with the object identity of the second object interacting with by the first object in the previous moment, it can be determined that an interaction object switching event has occurred for the first object.

[0037] In some embodiments, the electronic device 110 may, in response to detecting a change in a second object interacting with at least one first object, extract at least a portion of video segments from a plurality of video segments based on at least one frame in the video where the change was detected. For example, if a change is detected in the second object interacting with a certain first object between adjacent frames, the last frame in which the first object last interacted with the corresponding second object can be recorded as a potential segmentation point. The electronic device 110 may collect all potential segmentation points and construct a segmentation point set. Further, the electronic device 110 may divide the original video into a plurality of consecutive video segments 210 using the segmentation points in the segmentation point set as boundaries.

[0038] As an example, in a real-world video scenario, the first object can be a hand, and the second object can be an object that interacts with the hand. For instance, regarding the action of "opening box A with the right hand and then taking object B out of box A" in a video, the video can be segmented at the frame where box A is last manipulated by the right hand, thus generating two semantically continuous video segments. For example, the first video segment can correspond to the stage of "the right hand manipulating box A," and the second video segment can correspond to the stage of "the right hand manipulating object B." In this way, a stable interaction relationship between a single first object and its corresponding second object can be maintained within each video segment.

[0039] In some embodiments, multiple video segments 210 may be associated with video segment information 215. Video segment information 215 may include the starting frame index of each video segment and the number of frames contained in that video segment. For example, after video segmentation is completed, structured video segment information 215 may be output, where each video segment may contain information such as the starting frame index and the number of frames, for subsequent object location information determination processing within each video segment.

[0040] By segmenting the video based on the interaction relationship between the first and second objects, the interaction stages between objects can be explicitly represented, making each video segment correspond to a relatively independent interaction process. Compared to directly locating objects across the entire long video, this method reduces interference between different interaction stages, thus helping to ensure the consistency and continuity of object location results within each interaction cycle in the long video, and improving the stability and accuracy of subsequent video processing.

[0041] In some embodiments, the electronic device 110 may perform consistency optimization operations on the original detection and tracking results within each segmented video segment to improve the reliability of subsequent processing. For example, object category unification processing may be performed to determine a second identification result 220. The second identification result 220 may represent the detection and tracking results after category unification processing within each video segment, wherein the same object identity corresponds to a unique object category within the same video segment.

[0042] In some embodiments, objects with the same object identity may be identified as multiple categories within the same video segment (e.g., due to fluctuations in detection confidence, an object may be identified as a left-handed interactive object in some frames and a two-handed interactive object in others). In this case, if the second video segment is detected to include a second object of multiple categories, the electronic device 110 can determine the category of the second object in the video segment based on the frequency of occurrence of each category within the multiple categories in the video segment. The electronic device 110 can determine the category corresponding to the object's identifier as the category with the highest frequency of occurrence in the video segment. By unifying the object categories, the reliability of subsequent segmentation can be improved, and segmentation errors caused by fluctuations in detection confidence can be avoided.

[0043] In some embodiments, the category of the second object may include a predetermined category. The predetermined category can be a category used to represent a second object that interacts with multiple first objects simultaneously, for example, a category of objects that interact with multiple first objects at the same time. If the category of the multiple second objects is detected to be a predetermined category, the electronic device 110 can remove second objects from the multiple categories that correspond to other categories besides the predetermined category, thereby obtaining a third identification result 230. The third identification result 230 may represent the detection and tracking result after removing redundant objects and eliminating semantic conflicts between different object categories.

[0044] As an example, in a certain video segment, if the second recognition result 220 contains objects of the two-handed interactive object category, the electronic device 110 can remove the objects identified as corresponding one-handed interactive objects from the second recognition result 220, thereby avoiding semantic conflicts and improving the consistency of subsequent processing results.

[0045] Figure 3 A schematic diagram of a process 300 for determining location information according to some embodiments of the present disclosure is shown. Figure 3 As shown, after completing video segmentation and obtaining the third recognition result 230, the electronic device 110 can determine the position information of each of the multiple first objects and multiple second objects in each frame of the video based on multiple video segments. The third recognition result 230 can provide the category information, object identification, and bounding box information of each object in the video segment, thereby providing an initial reference for the subsequent determination of position information. For example, the electronic device 110 can first determine the local position information of the object in each video segment, and then integrate the local position information based on the temporal relationship between the video segments to obtain the position information of the object in the entire video range.

[0046] In some embodiments, for each of a plurality of first objects and a plurality of second objects, the electronic device 110 can determine the local position information of the object in each of the plurality of video segments based on a plurality of video segments. The local position information indicates the position of the object in each frame of the video segment. The local position information can indicate the position of the object in each frame of the corresponding video segment; for example, the local position information can be represented as a region range of the object in each frame or a pixel-level mask. By determining the object position information within a video segment, the relatively stable interaction relationship within the video segment can be utilized to improve the accuracy and continuity of object positioning.

[0047] In some examples, the process of determining the local location information of each object in each video segment is illustrated using an object in the first video segment out of multiple video clips as an example. (Reference) Figure 3 In box 310, electronic device 110 can acquire multiple frames from the first video segment, each of which contains the object. For example, for each interactive object in the first video segment, multiple start frames can be sampled at predetermined intervals in the frames where the object appears, where the number of samples is K, and K is an integer greater than or equal to 1. For example, a typical value of K can be 5. Alternatively or additionally, the start frames can be selected using random sampling. By sampling multiple start frames, multiple independent starting points can be provided for subsequent temporal propagation, thereby improving the stability and robustness of the object segmentation results.

[0048] In frame 320, the electronic device 110 can determine visual cues for the object in each frame across multiple frames. These visual cues can be used to represent the object's geometric features and to guide subsequent pixel-level segmentation or temporal propagation processes.

[0049] In some embodiments, the electronic device 110 can determine the corresponding mask of the object in multiple frames based on the bounding boxes of the object in each of the multiple frames. For example, the bounding boxes can be used as initial cues input to a pixel-level segmentation model to generate an initial binary mask of the object in the corresponding frame. The segmentation model here can be any suitable type of machine learning model that can be used for image segmentation. The initial binary mask obtained by the segmentation model can be understood as a relatively coarse binary mask.

[0050] Furthermore, the electronic device 110 can determine the geometric shape information of the object across multiple frames based on a corresponding mask. The geometric shape information can be used to characterize the spatial distribution features of the object region, such as the object's outline, key points, or circumscribed shape, thereby providing a basis for subsequently generating visual cues.

[0051] In some embodiments, the geometric information may include multiple target geometric points. The electronic device 110 can extract the foreground pixels of the object from corresponding masks. For example, pixels with a value of 1 (or a non-zero value) in corresponding masks of the object across multiple frames are considered foreground pixels. Further, the electronic device 110 can determine the bounding rectangle of the object based on the foreground pixels. For example, the minimum bounding rectangle enclosing the foreground pixels of the object can be calculated based on the extracted foreground pixels. This bounding rectangle can be an axially aligned rectangle or a rotated rectangle.

[0052] Furthermore, the electronic device 110 can determine multiple initial geometric points based on the circumscribed rectangle. For example, the four corner points of the circumscribed rectangle (e.g., top left, top right, bottom left, bottom right) and the center point of the circumscribed rectangle, for a total of five points, can be selected as initial geometric points. It is understood that the number of initial geometric points is not limited to five, and more or fewer points can be selected depending on the complexity of the object's shape, such as using a combination of corner points and side midpoints, or using uniformly sampled points.

[0053] In some embodiments, the electronic device 110 can determine the target geometric point corresponding to each initial geometric point based on the positional relationship between each pixel in the foreground pixels and each initial geometric point. For example, for each initial geometric point, the pixel with the closest Euclidean distance to the initial geometric point is found in the foreground pixel set, and this nearest neighbor pixel is used as the candidate target point corresponding to the initial geometric point. This operation can "pull" the geometric points on the circumscribed rectangle back into the actual object area, thereby better fitting the object shape.

[0054] To improve the stability of geometric points and avoid drastic changes in key point positions due to minor mask fluctuations, in some embodiments, the electronic device 110 can perform smoothing processing on the target geometric points. For example, for the target geometric points corresponding to the four corner points, the coordinates are averaged with those corresponding to the center point. This smoothing operation can cause the corner points to slightly shrink towards the center point, thereby improving the spatial stability of the geometric points. This method is particularly suitable for situations where there is noise at the object edges or the mask is incomplete.

[0055] In some embodiments, after position smoothing, the electronic device 110 can re-determine the corresponding foreground pixels near the smoothed geometric points to obtain multiple optimized geometric key points. These optimized geometric key points can collectively constitute the visual cue information for the object in the corresponding frame and serve as cue input for subsequent object segmentation or temporal propagation models. In this way, more stable geometric cue information can be generated based on the object bounding box, thereby improving object segmentation accuracy in complex shapes or under occlusion and reducing error propagation between video frames.

[0056] In some embodiments, the electronic device 110 can use multiple frames as starting frames and, based on visual cues of the object in the multiple frames, perform temporal propagation of the target position along the frame sequence in the first video segment to obtain multiple initial position information of the object in the multiple frames. For example, for each selected starting frame, the electronic device 110 uses the visual cues in the starting frame as guidance and employs any suitable video object segmentation model or mask propagation algorithm to perform temporal propagation along the video frame sequence in a forward and backward direction, thereby generating a complete mask sequence covering the entire first video segment. By performing temporal propagation independently with multiple starting frames, multiple prediction results for the same object can be obtained, thus providing a basis for subsequent result fusion.

[0057] In box 320, electronic device 110 can determine the local position information of the object in the first video segment based on multiple initial position information. For example, electronic device 110 can obtain more stable object position information by fusing multiple initial position information to reduce the error caused by a single temporal propagation.

[0058] As an example, the initial location information may include a mask sequence for the object. The mask sequence includes mask information corresponding to each frame in the first video segment. The electronic device 110 may sample multiple starting frames in the frames where the object appears, for example, by sampling at predetermined intervals or by random sampling, to obtain K starting frames, where K is an integer greater than or equal to 1. For example, a typical value for K could be 5. For each starting frame, the electronic device 110 may independently initiate a temporal propagation process based on the visual cue information corresponding to that starting frame, thereby obtaining a mask sequence corresponding to that starting frame. In this way, K independent mask sequences can be obtained.

[0059] In some embodiments, timing propagation may include forward propagation and backward propagation. For example, the electronic device 110 may, based on a selected starting frame, perform timing propagation of the target position along a forward frame sequence (e.g., from the starting frame to the end of the video clip) and simultaneously along a backward frame sequence (e.g., from the starting frame to the beginning of the video clip). This bidirectional propagation method fully utilizes the timing information in both directions before and after the starting frame, effectively covering frames preceding the starting frame, thereby improving the completeness of the segmentation results within the entire video clip.

[0060] In some embodiments, for each frame of a first video segment, the electronic device 110 can determine multiple mask information of the object in that frame from multiple mask sequences. Based on the multiple mask information, the target mask information of the object in that frame is determined. For example, for the same frame, K different mask sequences each yield a predicted mask. The electronic device 110 can determine the final target mask information of the object in that frame based on these K mask information using a fusion strategy.

[0061] In some embodiments, for each pixel in a plurality of mask information, the electronic device 110 may determine the number of mask information indicating that the pixel is a foreground pixel of the object; and determine the ratio of the number to the total number of the plurality of mask information. In response to the ratio exceeding a threshold ratio, the electronic device 110 may identify the pixel as a pixel in the target mask information. For example, the electronic device 110 may count the number of times the pixel is predicted to be foreground in K mask information and calculate the ratio between the number and K. If the ratio exceeds a preset threshold ratio, the electronic device 110 may determine that the pixel is foreground and include it in the target mask information; otherwise, it may determine it as background.

[0062] In some embodiments, the electronic device 110 can determine the local location information of the object in the first video segment based on the target mask information of the object in each frame of the first video segment. (Output the fused binary mask sequence. For example, the electronic device 110 can arrange the target mask information of all frames in frame order to form the local location information of the object in the video segment (for example, the local location information can be a continuous binary mask sequence).

[0063] By using the above method, mask sequences generated by multiple starting frames can be fused, thereby effectively reducing the sensitivity of single-path temporal propagation to occlusion, rapid motion, or local errors, and significantly improving the temporal continuity and anti-interference ability of object segmentation results.

[0064] Continue to refer to Figure 3 In box 340, electronic device 110 can determine the object's location information based on the object's local location information in each video segment and the temporal relationship between multiple video segments. For example, electronic device 110 can integrate the local location information of the object determined in each video segment according to the temporal order of multiple video segments in the original video, thereby obtaining the continuous location information of the object in the entire video.

[0065] In some embodiments, the electronic device 110 can uniformly organize the local location information of multiple objects in multiple video segments and combine it with the category information of each object to generate a semantic segmentation annotation sequence for the video. The semantic segmentation annotation sequence can be used to represent the pixel regions corresponding to different objects in each frame of the video.

[0066] As an example, electronic device 110 can generate a corresponding semantic segmentation annotation image for each frame of a video, where pixel values ​​in the image represent different semantic categories. For example, different pixel values ​​can represent background regions and regions corresponding to different object categories. In some examples, pixel values ​​can be used to distinguish categories such as background, left hand, right hand, left-hand interactive object, right-hand interactive object, and two-hand interactive object. In some examples, the semantic segmentation annotation sequence can be organized and stored according to the frame sequence, so that each frame corresponds to a semantic annotation result, thereby forming complete video semantic segmentation annotation data. In this way, object location information obtained from multiple video segments can be uniformly mapped to the complete video, thereby obtaining semantic segmentation results with continuous temporal structure.

[0067] In summary, according to the embodiments of this disclosure, by analyzing the interaction relationship between the first object and the second object in the video, and combining the interaction relationship-driven video segmentation, geometrically guided visual cues generation, and multi-path temporal segmentation and fusion mechanism, the stable determination of the object position information in the video can be achieved, thereby improving the accuracy and temporal consistency of video object segmentation results in complex interactive scenarios.

[0068] On one hand, an interaction-aware segmentation mechanism based on changes in the interaction relationship between the first and second objects analyzes the target tracking results and automatically divides the video into segments based on the switching events of the second object interacting with the first object. In this way, each video segment corresponds to a complete interaction process, ensuring that the interaction relationship between the first and second objects in each segment remains relatively stable. Through this method, the temporal structure of object interaction relationships can be explicitly modeled, thereby reducing segmentation interference between different interacting objects and improving the stability of segmentation results in object interaction scenarios in long videos.

[0069] On the other hand, through a geometry-guided visual cue generation mechanism, the geometric structure information of the object is extracted from the coarse object segmentation mask, and multiple spatial key points are determined based on the geometric structure information, thereby generating visual cue information for object segmentation or temporal propagation. By optimizing the initial geometric points, key point cues with stable spatial distribution that can reflect the shape characteristics of the object can be obtained, thereby improving the adaptability of the initial cues to complex deformations, occlusions, and irregular shapes. In some embodiments, the number of key points is not limited to a specific number; multiple key points that can characterize the object shape can be determined according to the object shape characteristics through random sampling, clustering of key points, or sampling of the farthest point.

[0070] Furthermore, a multi-path temporal fusion segmentation mechanism is employed, triggering the temporal propagation process of object segmentation with multiple starting frames and fusing the segmentation results obtained from multiple propagation paths. For example, a pixel-level mask fusion strategy can be used to comprehensively evaluate the prediction results of multiple propagation paths, thereby obtaining a more stable segmentation result. This approach effectively reduces the sensitivity of single-path propagation to occlusion, rapid motion, or local errors, and significantly improves the temporal continuity and anti-interference capability of video object segmentation results.

[0071] Figure 4 A flowchart of a process 400 for video processing according to some embodiments of the present disclosure is shown. Process 400 may be implemented in environment 100, for example, process 400 may be implemented at electronic device 110.

[0072] In frame 410, electronic device 110 acquires video, which includes a plurality of first objects and a plurality of second objects.

[0073] In box 420, electronic device 110 extracts multiple video segments from the video based on the interaction relationship between the plurality of first objects and the plurality of second objects, wherein in each of the plurality of video segments, at least one first object interacts with a second object.

[0074] In frame 430, electronic device 110 determines the position information of the plurality of first objects and the plurality of second objects in each frame of the video based on the plurality of video segments.

[0075] In some embodiments, the interaction relationship is determined by: determining the corresponding object identifiers and bounding boxes of the plurality of first objects and the plurality of second objects in each frame of the video; and determining the interaction relationship based on the corresponding object identifiers and bounding boxes.

[0076] In some embodiments, extracting multiple video segments from a video includes: detecting whether a second object interacting with each first object in the video has changed; and in response to detecting a change in a second object interacting with at least one first object, extracting at least a portion of the multiple video segments from the video based on at least one frame in the video where the change was detected.

[0077] In some embodiments, determining location information includes: for each of a plurality of first objects and a plurality of second objects, determining local location information of the object in each of the plurality of video segments based on a plurality of video segments, wherein the local location information indicates the position of the object in each frame of the video segment; and determining the location information of the object based on the determined local location information and the temporal relationship between the plurality of video segments.

[0078] In some embodiments, determining local location information includes: for a first video segment among multiple video segments, obtaining multiple frames from the first video segment, each of the multiple frames containing the object; using the multiple frames as starting frames, and based on visual cues of the object in each of the multiple frames, performing temporal propagation of the target position along the frame sequence in the first video segment to obtain multiple initial location information of the object in the multiple frames, wherein the visual cues are used to represent the geometric features of the object; and determining local location information of the object in the first video segment based on the multiple initial location information.

[0079] In some embodiments, the initial location information includes a mask sequence for the object, the mask sequence including mask information corresponding to each frame in the first video segment, and determining the local location information of the object in the first video segment includes: for each frame in the first video segment, determining multiple mask information of the object in that frame in multiple mask sequences; determining target mask information of the object in that frame based on the multiple mask information; and determining the local location information of the object in the first video segment based on the target mask information of the object in each frame of the first video segment.

[0080] In some embodiments, visual cue information is determined by: determining a corresponding mask for the object in each of the multiple frames based on the bounding box of the object in each of the multiple frames; determining the geometric shape information of the object in the multiple frames based on the corresponding mask; and determining visual cue information based on the geometric shape information.

[0081] In some embodiments, timing transfer includes: timing transfer of the target position along a forward frame sequence based on a start frame; and timing transfer of the target position along a reverse frame sequence based on a start frame.

[0082] In some embodiments, determining target mask information includes: for each pixel in a plurality of mask information, determining the number of mask information indicating that the pixel is a foreground pixel of the object; determining a ratio of the number to the total number of the plurality of mask information; and determining the pixel as a pixel in the target mask information in response to the ratio exceeding a threshold ratio.

[0083] In some embodiments, the geometric shape information includes target geometric points, and determining the geometric shape information includes: extracting foreground pixels of the object from a corresponding mask; determining the bounding rectangle of the object based on the foreground pixels; determining a plurality of initial geometric points based on the bounding rectangle; and determining the target geometric point corresponding to each initial geometric point based on the positional relationship between each pixel in the foreground pixels and each initial geometric point.

[0084] In some embodiments, process 400 further includes: detecting, for a second video segment among a plurality of video segments, that the second video segment includes a second object of a plurality of categories; and determining the category of the second object in the video segment based on the frequency of occurrence of each category among the plurality of categories in the video segment.

[0085] In some embodiments, the second object category includes a predetermined category, and the process 400 further includes: detecting that a plurality of categories include the predetermined category; and removing second objects from the second objects of the plurality of categories that correspond to other categories besides the predetermined category.

[0086] In some embodiments, the first object is a hand, and the category of the second object includes at least one of the following: a left-hand interactive object, a right-hand interactive object, or a two-hand interactive object.

[0087] Figure 5 A block diagram of an apparatus 500 for video processing according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or include an electronic device 110.

[0088] The apparatus 500 includes an acquisition module 510 configured to acquire a video, the video including a plurality of first objects and a plurality of second objects; an extraction module 520 configured to extract a plurality of video segments from the video based on the interaction relationships between the plurality of first objects and the plurality of second objects, wherein in each of the plurality of video segments at least one first object interacts with a second object; and a determination module 530 configured to determine, based on the plurality of video segments, the position information of each of the plurality of first objects and the plurality of second objects in each frame of the video.

[0089] In some embodiments, the interaction relationship is determined by: determining the corresponding object identifiers and bounding boxes of the plurality of first objects and the plurality of second objects in each frame of the video; and determining the interaction relationship based on the corresponding object identifiers and bounding boxes.

[0090] In some embodiments, the extraction module 520 is further configured to: detect whether a second object interacting with each first object in the video has changed; and in response to detecting a change in a second object interacting with at least one first object, extract at least a portion of video segments from a plurality of video segments in the video based on at least one frame in the video where the change was detected.

[0091] In some embodiments, the determining module 530 is further configured to: for each of the plurality of first objects and the plurality of second objects, determine local position information of the object in each of the plurality of video segments based on the plurality of video segments, wherein the local position information indicates the position of the object in each frame of the video segment; and determine the position information of the object based on the determined local position information and the temporal relationship between the plurality of video segments.

[0092] In some embodiments, the determining module 530 is further configured to: for a first video segment among a plurality of video segments, obtain a plurality of frames from the first video segment, each of the plurality of frames containing the object; using the plurality of frames as starting frames, based on visual cue information of the object in each of the plurality of frames, perform temporal transmission of the target position along the frame sequence in the first video segment to obtain a plurality of initial position information of the object in the plurality of frames, wherein the visual cue information is used to represent the geometric features of the object; and determine the local position information of the object in the first video segment based on the plurality of initial position information.

[0093] In some embodiments, the initial location information includes a mask sequence for the object, the mask sequence including mask information corresponding to each frame in the first video segment, and the determining module 530 is further configured to: for each frame in the first video segment, determine multiple mask information of the object in that frame in the multiple mask sequences; determine target mask information of the object in that frame based on the multiple mask information; and determine local location information of the object in the first video segment based on the target mask information of the object in each frame in the first video segment.

[0094] In some embodiments, visual cue information is determined by: determining a corresponding mask for the object in each of the multiple frames based on the bounding box of the object in each of the multiple frames; determining the geometric shape information of the object in the multiple frames based on the corresponding mask; and determining visual cue information based on the geometric shape information.

[0095] In some embodiments, timing transfer includes: timing transfer of the target position along a forward frame sequence based on a start frame; and timing transfer of the target position along a reverse frame sequence based on a start frame.

[0096] In some embodiments, the determining module 530 is further configured to: for each pixel in the plurality of mask information, determine the number of mask information indicating that the pixel is a foreground pixel of the object; determine the ratio of the number to the total number of the plurality of mask information; and determine the pixel as a pixel in the target mask information in response to the ratio exceeding a threshold ratio.

[0097] In some embodiments, the geometric shape information includes target geometric points, and the determining module 530 is further configured to: extract the foreground pixels of the object from the corresponding mask; determine the bounding rectangle of the object based on the foreground pixels; determine a plurality of initial geometric points based on the bounding rectangle; and determine the target geometric point corresponding to each initial geometric point based on the positional relationship between each pixel in the foreground pixels and each initial geometric point.

[0098] In some embodiments, the apparatus 500 further includes a category determination module configured to: detect, for a second video segment among a plurality of video segments, that the second video segment includes a second object of a plurality of categories; and determine the category of the second object in the video segment based on the frequency of occurrence of each of the plurality of categories in the video segment.

[0099] In some embodiments, the second object category includes a predetermined category, and the apparatus 500 further includes a category determination module configured to: detect that a plurality of categories include the predetermined category; and remove second objects from the second objects of the plurality of categories that correspond to other categories among the plurality of categories besides the predetermined category.

[0100] In some embodiments, the first object is a hand, and the category of the second object includes at least one of the following: a left-hand interactive object, a right-hand interactive object, or a two-hand interactive object.

[0101] The modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 500 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0102] Figure 6 A block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0103] like Figure 6As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0104] Electronic device 600 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 600.

[0105] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0106] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or another network node.

[0107] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interfaces (not shown).

[0108] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above.

[0109] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable video processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable video processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable video processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0111] Computer-readable program instructions may also be loaded onto a computer, other programmable video processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable video processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable video processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0113] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for video processing, comprising: Acquire a video, wherein the video includes multiple first objects and multiple second objects; Based on the interaction relationship between the plurality of first objects and the plurality of second objects, a plurality of video segments are extracted from the video, and in each of the plurality of video segments, at least one first object interacts with a second object. as well as Based on the multiple video segments, determine the position information of the multiple first objects and the multiple second objects in each frame of the video.

2. The method of claim 1, wherein the interaction relationship is determined in the following manner: Determine the corresponding object identifiers and bounding boxes of the plurality of first objects and the plurality of second objects in each frame of the video; and The interaction relationship is determined based on the corresponding object identifier and bounding box.

3. The method of claim 1, wherein extracting multiple video segments from the video comprises: Detect whether the second object that interacts with each first object in the video has changed; as well as In response to the detection of a change in a second object that interacts with at least one first object, at least a portion of the video segments are extracted from the video based on at least one frame in the video in which the change was detected.

4. The method according to claim 1, wherein determining the location information includes: For each of the plurality of first objects and the plurality of second objects, Based on the plurality of video segments, local position information of the object in each of the plurality of video segments is determined, wherein the local position information indicates the position of the object in each frame of the video segment; as well as The location information of the object is determined based on the determined local location information and the temporal relationship between the multiple video segments.

5. The method according to claim 4, wherein determining the local location information includes: For the first video segment among the plurality of video segments, Multiple frames are obtained from the first video segment, and each of the multiple frames contains the object; Using the plurality of frames as starting frames, and based on the visual cue information of the object in each of the plurality of frames, the target position is sequentially transmitted along the frame sequence in the first video segment to obtain multiple initial position information of the object in the plurality of frames. The visual cue information is used to represent the geometric features of the object. Based on the multiple initial location information, the local location information of the object in the first video segment is determined.

6. The method of claim 5, wherein the initial location information includes a mask sequence for the object, the mask sequence including mask information corresponding to each frame in the first video segment, and determining the local location information of the object in the first video segment includes: For each frame in the first video segment, Determine multiple mask information of the object in the frame from the multiple mask sequences; Based on the multiple mask information, the target mask information of the object in the frame is determined; as well as Based on the target mask information of the object in each frame of the first video segment, the local location information of the object in the first video segment is determined.

7. The method of claim 5, wherein the visual cue information is determined by: Based on the bounding box of the object in each of the multiple frames, determine the corresponding mask of the object in the multiple frames; Based on the corresponding mask, the geometric shape information of the object in the multiple frames is determined; as well as The visual cue information is determined based on the geometric shape information.

8. The method of claim 5, wherein the timing transfer comprises: Based on the starting frame, the target position is transferred in a time sequence along the forward frame sequence; as well as Based on the starting frame, the target position is transferred in a time sequence along the reverse frame sequence.

9. The method of claim 6, wherein determining the target mask information comprises: For each pixel in the plurality of mask information, determine the number of mask information that indicates that the pixel is a foreground pixel of the object; Determine the proportion of the number relative to the total number of the plurality of mask information; as well as In response to the ratio exceeding a threshold ratio, the pixel is identified as a pixel in the target mask information.

10. The method of claim 7, wherein the geometric shape information includes target geometric points, and determining the geometric shape information includes: Extract the foreground pixels of the object from the corresponding mask; Based on the foreground pixels, determine the bounding rectangle of the object; Based on the circumscribed rectangle, multiple initial geometric points are determined; as well as Based on the positional relationship between each pixel in the foreground pixels and each initial geometric point, the target geometric point corresponding to each initial geometric point is determined.

11. The method according to claim 1, further comprising: Regarding the second video segment among the plurality of video segments, The second video segment was detected to include second objects of multiple categories; as well as The category of the second object in the video segment is determined based on the frequency of occurrence of each of the multiple categories in the video segment.

12. The method of claim 11, wherein the second object category includes a predetermined category, and the method further comprises: The multiple categories were detected to include the predetermined category; as well as Remove the second objects from the plurality of categories that correspond to other categories besides the predetermined category.

13. The method according to claim 1, wherein the first object is a hand. The category of the second object includes at least one of the following: left-hand interactive object, right-hand interactive object, or two-hand interactive object.

14. An apparatus for object detection, comprising: The acquisition module is configured to acquire a video, the video including a plurality of first objects and a plurality of second objects; The extraction module is configured to extract multiple video segments from the video based on the interaction relationship between the plurality of first objects and the plurality of second objects, wherein in each of the plurality of video segments, at least one first object interacts with one second object. as well as The determination module is configured to determine the position information of each of the plurality of first objects and the plurality of second objects in each frame of the video, based on the plurality of video segments.

15. An electronic device comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 13.

17. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 13.

Images