Image sequence segmentation method, device and computer program product

Abstract

The present disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for segmenting an image sequence, involving the fields of deep learning, large models, and other artificial intelligence technologies, and can be applied in smart sports scenarios. The specific implementation scheme comprises: determining the motion state of a target object in the image sequence based on images in the image sequence to obtain a motion state sequence; updating the target motion state among the multiple motion states based on multiple motion states in a sliding window in the motion state sequence to obtain an updated motion state that conforms to the kinematic laws of the target object; determining a segmentation point corresponding to a motion process of the target object in the image sequence based on the updated motion state; and segmenting the image sequence based on the segmentation point.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology fields such as deep learning and large models, and in particular to an image sequence segmentation method, device, electronic device, storage medium, and computer program product, which can be applied to scenarios such as smart sports. Background Art

[0002] In today's professional sports, there's a growing demand for high-precision, real-time performance analysis. This demand stems from the constant pursuit of optimizing athlete performance, including precise analysis of technical details, immediate adjustments to athletic strategies, and effective management of injury prevention. Traditional analysis methods often rely on manual observation of sports videos and post-processing of the data. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for segmenting an image sequence.

[0004] According to a first aspect, a method for segmenting an image sequence is provided, comprising: determining, based on images in the image sequence, a motion state of a target object in the image to obtain a motion state sequence; updating, based on a plurality of motion states in a sliding window in the motion state sequence, a target motion state among the plurality of motion states to obtain an updated motion state that conforms to the kinematic law of the target object; determining, based on the updated motion state, a segmentation point corresponding to a motion process of the target object in the image sequence; and segmenting the image sequence according to the segmentation point.

[0005] According to a second aspect, a device for segmenting an image sequence is provided, comprising: a state sequence determination unit, configured to determine, based on an image in the image sequence, a motion state of a target object in the image, and obtain a motion state sequence; a state update unit, configured to update, according to a plurality of motion states in a sliding window in the motion state sequence, a target motion state among a plurality of motion states, and obtain an updated motion state that conforms to the kinematic law of the target object; a segmentation point determination unit, configured to determine, according to the updated motion state, a segmentation point corresponding to a motion process of the target object in the image sequence; and an image sequence segmentation unit, configured to segment the image sequence according to the segmentation point.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to execute the method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] According to the technology disclosed in the present invention, a method and device for segmenting an image sequence are provided. According to multiple motion states in a sliding window in a motion state sequence, a target motion state is updated so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations during the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and further improving the real-time performance and accuracy of the segmentation process for the image sequence corresponding to the motion state sequence.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0013] Figure 2 is a flowchart of an embodiment of a method for segmenting an image sequence according to the present disclosure;

[0014] Figure 3 Schematic diagram of image acquisition during trampoline motion according to this embodiment;

[0015] Figure 4 is a schematic diagram of a specific determination process of the updated motion state according to this embodiment

[0016] Figure 5 is a schematic diagram of an application scenario of the image sequence segmentation method according to this embodiment;

[0017] Figure 6 is a schematic diagram of a display interface of a target display device according to this embodiment;

[0018] Figure 7 is a flow chart of an embodiment of a data recommendation method according to the present disclosure;

[0019] Figure 8is a structural diagram of an embodiment of an image sequence segmentation device according to the present disclosure;

[0020] Figure 9 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 An exemplary architecture 100 is shown to which the image sequence segmentation method and apparatus disclosed herein can be applied.

[0024] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 constitute a topological network, and network 104 is used to provide a medium for communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0025] Terminal devices 101, 102, and 103 can be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing, and other functions, including but not limited to smartphones, tablet computers, e-book readers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, for example, to provide distributed services, or they can be implemented as a single software or software module. No specific limitations are given here.

[0026] Server 105 can be a server that provides various services, such as a backend processing server that determines segmentation points for the image sequence provided by terminal devices 101, 102, and 103 to segment the image sequence. Optionally, the server can feed back a segmented sequence obtained by segmenting the image sequence to the terminal device. As an example, server 105 can be a cloud server.

[0027] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0028] It should also be noted that the image sequence segmentation method provided in the embodiments of the present disclosure is generally executed by a server, but the possibility of execution by a terminal device, or the server and terminal device cooperating with each other, is not excluded. Accordingly, the various components (e.g., various units) included in the image sequence segmentation device can be entirely provided in the server, entirely provided in the terminal device, or separately provided in the server and the terminal device.

[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. When the electronic device on which the image sequence segmentation method is running does not need to transmit data to other electronic devices, the system architecture may only include the electronic device (e.g., terminal device or server) on which the image sequence segmentation method is running.

[0030] Please refer to Figure 2 , Figure 2 This is a flow chart of a method for segmenting an image sequence provided by an embodiment of the present disclosure. In process 200, the following steps are included:

[0031] Step 201: determining the motion state of a target object based on images in an image sequence to obtain a motion state sequence.

[0032] In this embodiment, the execution subject of the image sequence segmentation method (for example, Figure 1 The server in the image sequence can obtain the image sequence remotely or locally through a wired network connection or a wireless network connection, and determine the motion state of the target object in the image based on the images in the image sequence to obtain a motion state sequence.

[0033] An image sequence is a raw video recorded with the target subject's authorization, showing the target subject in motion, or a processed video obtained by subjecting the raw video to specific processing (e.g., enhancing clarity, filtering key frames, etc.). The target subject can be a moving object, such as an athlete or various sports equipment operated by the athlete.

[0034] As an example, the execution entity may input the images in the image sequence into the motion state determination model one by one or in batches, determine the motion state of the target object in each image in the image sequence through the motion state determination model, and combine the motion states of the target objects in each image according to the temporal relationship represented by the image sequence to obtain a motion state sequence. The motion state determination model is used to represent the correspondence between the image and the motion state of the target object in the image, and can be, for example, a convolutional neural network, a recurrent neural network, a large visual language model, etc.

[0035] As another example, for each image in an image sequence, the above-mentioned execution entity can determine the feature information (such as facial features, limb features) of the target object in the image, and then determine the motion state of the target object in the image based on the similarity between the feature information and the standard feature information of each motion state.

[0036] The target object's motion state varies depending on the sport. For example, in jumping sports like trampoline, pole vaulting, and high jump, the motion state includes ascending and descending. In ball sports like football and basketball, the motion state includes holding (touching) and not holding (not touching).

[0037] In some optional implementations of this embodiment, the images in the image sequence are depth images. A depth image, also known as a range image, is an image that uses the distance (depth) from an image acquisition device (such as a camera or depth sensor) to each point in the scene as its pixel value.

[0038] In this implementation, the execution entity may perform step 201 as follows:

[0039] First, a target depth image corresponding to the motion range of the target object is determined from the depth image.

[0040] In this implementation, the upper depth threshold Dmax and the lower depth threshold Dmin are determined in advance according to the actual situation on site; then the pixels with pixel values ​​between the lower threshold Dmin and the upper threshold Dmax are determined in the depth image to obtain the target depth image corresponding to the motion range of the target object. Figure 3, shows a schematic diagram of image acquisition during trampoline exercise. In trampoline exercise, the athlete's movement process is entirely on the trampoline, meaning their range of motion is the area corresponding to the trampoline. Therefore, the distance between the image acquisition device and the near end of the trampoline (the end closer to the image acquisition device) can be used as the lower threshold Dmin, and the distance between the image acquisition device and the far end of the trampoline (the end farther from the image acquisition device) can be used as the upper threshold Dmax.

[0041] Then, the detection box corresponding to the target object is determined from the target depth image.

[0042] The detection frame is the minimum bounding box corresponding to the target object. The above execution entity can binarize the target depth image to determine the detection frame corresponding to the target object.

[0043] Finally, the motion state of the target object is determined based on the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

[0044] As an example, the two-dimensional center coordinate Pos of the target object can be obtained by the center of the detection box BBox coordinate; now The two-dimensional center coordinate Pos at the moment (corresponding to the target depth image) t and T now-1 Time position Pos t-1 , the displacement is calculated by the following formula:

[0045] ΔY=(Pos t -Pos t-1 )×Y

[0046] Wherein, Y represents the height. When ΔY>0, it is in an ascending state; when ΔY<0, it is in a descending state.

[0047] In this implementation, a specific method for determining the motion state of a target object in an image is provided, which is closely adapted to the acquisition method and improves the efficiency of determining the motion state.

[0048] Step 202 : updating a target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence, and obtaining an updated motion state that conforms to the kinematic law of the target object.

[0049] In this embodiment, the execution subject may update the target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence, and obtain an updated motion state that conforms to the kinematic law of the target object.

[0050] The capacity of the sliding window can be set according to actual conditions, for example, the capacity is 5. The target motion state can be any one of multiple motion states in the sliding window, but the positions of the target motion states in different sliding windows are the same, for example, the target motion state of the current sliding window and the target motion state of the previous sliding window are both in the middle position of the sliding window.

[0051] Kinematic laws, derived through observation and experimentation, describe the regularity with which the position of a person or object in space changes over time. Taking trampoline exercise as an example, the five motion states in the sliding window are (ascending state, ascending state, descending state, ascending state, descending state). It's understandable that an athlete's ascending and descending states both persist for a certain period of time and don't change repeatedly in a short period of time. If these five motion states indicate that the target object's motion state changes repeatedly in a very short period of time, then one of the five motion states in the sliding window doesn't conform to kinematic laws.

[0052] Here's why the motion state doesn't conform to kinematic laws: Continuing with the example of trampoline exercise, the actual motion state changes typically involve the athlete ascending from the lowest point to the highest point and descending from the highest point to the lowest point. However, at the highest point, the athlete remains relatively still (at constant altitude) for a short period (e.g., 0.1 seconds). During this period, the video capture device typically captures multiple frames. For example, an image capture device with a frame rate of 50 captures one frame every 0.02 seconds, thus capturing five frames in 0.1 seconds. When determining the detection frame for the target object within these five frames, detection errors may result in inconsistent positions of the determined detection frame, resulting in fluctuations in the athlete's ascent and descent at the highest point. This situation is also known as short-term fluctuation interference.

[0053] As an example, the above-mentioned execution entity can determine whether the target motion state among multiple motion states conforms to the kinematic laws based on the multiple motion states in the sliding window; in response to the determination of yes, the target motion state is used as the updated motion state; in response to the determination of no, the target motion state is adjusted to conform to the kinematic laws based on the multiple motion states to obtain the adjusted motion state.

[0054] As another example, for different sports, the execution subject or an electronic device communicatively connected to the execution subject is provided with preset conditions representing the kinematic laws corresponding to the sports. The execution subject can determine the preset conditions corresponding to the sports represented by the image sequence, and then, based on the preset conditions and multiple motion states within the sliding window, determine whether the target motion state meets the preset conditions. If it meets the preset conditions, the target motion state is used as the updated motion state. If it does not meet the preset conditions, the target motion state is adjusted based on the multiple motion states to meet the preset conditions, thereby obtaining an adjusted motion state.

[0055] It can be understood that the sliding window slides on the motion sequence, and for each sliding window, an updated motion state of the target motion state within the sliding window will be obtained, thereby obtaining multiple updated motion states; multiple updated motion states are arranged according to the corresponding time sequence relationship of the motion state sequence to form an updated motion state.

[0056] In some optional implementations of this embodiment, the execution entity may perform step 202 as follows:

[0057] First, it is determined whether the number of motion states that are the same as the target motion state among the plurality of motion states exceeds a preset number threshold.

[0058] As an example, whether the number of motion states identical to the target motion state in the multiple motion states exceeds a preset number threshold is used to indicate whether the number of motion states identical to the target motion state in the multiple motion states exceeds half.

[0059] Then, in response to the determination being yes, it is determined whether taking the target motion state as the updated motion state complies with the kinematic law based on the historical updated motion state.

[0060] The historical updated motion state is an updated motion state determined according to the motion state in the historical sliding window.

[0061] In this implementation, the execution entity may determine updated motion states corresponding to all historical sliding windows up to the current time, or to a preset number of historical sliding windows up to the current time, i.e., historical updated motion states, to obtain a historical updated motion state sequence. Furthermore, based on the historical updated motion state sequence, it is determined whether using the target motion state as the updated motion state complies with kinematic laws.

[0062] Continuing with the trampoline exercise as an example, with the target motion state as the updated motion state, it is determined whether a short-term fluctuation occurs, thereby judging whether the target motion state as the updated motion state conforms to the kinematic law.

[0063] Finally, based on the judgment result being yes, the target motion state is determined as the updated motion state.

[0064] When it is determined that taking the target motion state as the updated motion state complies with the kinematic law, the target motion state is taken as the updated motion state.

[0065] In this implementation, a specific method for determining the updated motion state is provided, which improves the accuracy of the determined updated motion state.

[0066] In some optional implementations of this embodiment, the updated motion state includes an ascending state and a descending state corresponding to a jumping motion, and the kinematic law is characterized by preset conditions, which include:

[0067] The target motion state is different from the updated motion state corresponding to the previous sliding window; and the difference between the capture time of the image corresponding to the target motion state and the time of the previous state change process exceeds a preset time difference threshold. The state change process represents the process of change between adjacent updated motion states.

[0068] For jumping motion, the moment of short-term fluctuation interference is generally when the target object is at its highest or lowest point. At this time, the actual motion state of the target object will generally change, and the difference between the time of this change and the previous state change process will be large.

[0069] Taking the highest point as an example, at this time, a change from an ascending state to a descending state generally occurs. The moment when this state change process occurs, that is, the moment when the target object is at the highest point during its movement, can generally be represented by the acquisition moment of the image corresponding to the highest point. It should be separated from the moment when the state changes from an ascending state to a descending state in the previous trampoline process by a long time (exceeding the preset time difference threshold).

[0070] The preset time difference threshold can be set according to actual conditions and is not limited here.

[0071] In this implementation, for jumping motions, preset conditions for characterizing kinematic laws are specifically provided, which helps to improve the efficiency and accuracy of the process of determining whether the updated motion state conforms to the kinematic laws based on the target motion state.

[0072] In some optional implementations of this embodiment, the above-mentioned execution entity can also perform the above-mentioned step 202 in the following manner: in response to the number of motion states that are the same as the target motion state in multiple motion states not exceeding the preset number threshold, the motion state in multiple motion states whose number exceeds the preset number threshold is determined as the updated motion state.

[0073] For example, in a trampoline exercise, the five states in the sliding window are (ascending, ascending, descending, ascending, descending). The target state is the "descending" state in the middle, and its number is two. This does not exceed the preset threshold. Therefore, the "ascending" state that exceeds the threshold is used as the updated state corresponding to the target state.

[0074] In this implementation, a method for determining the updated motion state when the number does not exceed a preset threshold is provided, thereby improving the accuracy and determination efficiency of the updated motion state.

[0075] In some optional implementations of this embodiment, the target motion state is the last motion state among the multiple motion states.

[0076] In this implementation, the execution subject may perform step 202 in the following manner: updating the last motion state among the multiple motion states according to the multiple motion states to obtain an updated motion state.

[0077] In this implementation, the target motion state is the last motion state among the multiple motion states. The updated motion state can be obtained by determining the last motion state among the multiple motion states according to the multiple motion states through the various determination methods of the updated motion state mentioned above.

[0078] In this implementation, the target motion state is set to the last motion state among multiple motion states, and the updated motion state corresponding to the last motion state is determined based on the multiple motion states, which helps to further improve the accuracy of the updated motion state.

[0079] In some optional implementations of this embodiment, before executing step 202, the execution entity may further perform the following operation: determining whether the number of motion states in the sliding window reaches the capacity of the sliding window.

[0080] That is, when the number of motion states in the sliding window reaches the capacity of the sliding window, the operation of determining the updated motion state is not performed.

[0081] Taking the sliding window capacity of 5 as an example, in the initial stage, the first to fifth frames of the image sequence are sequentially sent into the sliding window. Until the fifth frame, the above-mentioned execution subject will execute the above-mentioned step 202 to obtain the updated motion state; corresponding to the first four frames of the image, the above-mentioned execution subject will not execute the above-mentioned step 202.

[0082] In this implementation, the determination operation of the updated motion state is performed only when the number of motion states in the sliding window accumulates to the capacity of the sliding window, which helps to further improve the accuracy of the updated motion state.

[0083] Continue to refer Figure 4 , which shows a schematic diagram of a specific process for determining the updated motion state.

[0084] The input signal of the determination system that defines the updated motion state (motion state in the motion state sequence) S in And the output signal (updated motion state) S out Both are Boolean variables, which have only two possibilities: positive and negative, corresponding to rising and falling states respectively. The sliding window length is N, where N is an odd number.

[0085] 1. Initialization: In the initial stage of motion analysis, the system has no existing state. Whenever the input signal S in , accumulated signal S in To the sliding window, no signal is output before N input signals are accumulated. When N frames are accumulated, S stable Set the signal to be greater than N / 2 and set the time when the state change process occurs T stable is 0.

[0086] 2. Main loop: After the input signal accumulates to N, the sliding window continues to slide along the motion state sequence. Whenever the input signal S in , perform the following operations:

[0087] 2.1、If there are N motion states in the sliding window, and the signal S in If the number of the same motion states does not exceed N / 2, the system is in an "unstable state" and the signal S stable Determined to be S in The corresponding updated motion state.

[0088] 2.2 If the N motion states within the sliding window are within the same range as the signal S in If the number of the same motion states exceeds N / 2, the system is in a "stable state" and the current S stable ; if S in Not equal to S stable And the current time T now Distance T stable The time is greater than the preset time difference threshold T th , then the signal S stable Updated to S in ,T stable Set to T now , and S in As the updated motion state.

[0089] Step 203: Determine a segmentation point corresponding to a motion process of the target object in the image sequence according to the updated motion state.

[0090] In this embodiment, the execution subject can update the post-motion state and determine the segmentation points corresponding to a motion process of the target object in the image sequence. The image sequence often represents the repeated motion process of the target object, and the image sequence between adjacent segmentation points represents a motion process of the target object.

[0091] As an example, the execution entity may analyze each updated motion state in the updated motion state sequence to determine the start time and the end time of a motion process of the target object; and then use the start time and the end time as segmentation points.

[0092] As another example, the execution entity can determine the corresponding segmentation point determination method for different sports. For example, for jumping sports, the segmentation point is the transition between the ascending and descending states; for ball sports, the segmentation point is the transition between the holding and non-holding states. In this way, the execution entity can determine the segmentation point corresponding to a movement of the target object in the image sequence based on the segmentation point determination method corresponding to the sports represented by the updated motion state sequence and image sequence.

[0093] It can be understood that the images in the image sequence and the motion states in the motion state sequence are in one-to-one correspondence; after the number of motion states in the sliding window accumulates to the capacity of the sliding window, the motion states in the motion state sequence and the updated motion states in the updated motion state sequence are in one-to-one correspondence.

[0094] In some optional implementations of this embodiment, the above-mentioned execution entity can perform the above-mentioned step 203 in the following manner: for an image in an image sequence, in response to the updated motion state corresponding to the image being a descending state, and the updated motion state corresponding to the previous frame image of the image being an ascending state, a segmentation point is determined between the image and the previous frame image of the image.

[0095] Continue to refer to the above Figure 4 The corresponding implementation method, when S occurs t ! =S t-1 , it indicates that the jump state has changed:

[0096] If S t If the target object is in the descending state, then the target object is changing from the ascending state to the descending state. now At the highest point of the jump;

[0097] If S t If the target object is in the rising state, it is changing from the falling state to the rising state. now At the lowest point of the jump, this jump ends and the next jump begins.

[0098] In this implementation, a segmentation point determination method for jumping motions is provided, which improves the accuracy and efficiency of the segmentation point determination operation.

[0099] Step 204: segment the image sequence according to the segmentation points.

[0100] In this embodiment, the execution entity may segment the image sequence according to segmentation points.

[0101] As an example, the execution subject segments the image sequence according to the segmentation points to obtain multiple segmented sequences, each of which represents a motion process of the target object.

[0102] Continue to see Figure 5 , Figure 5 FIG5 is a schematic diagram 500 illustrating an application scenario of the image sequence segmentation method according to this embodiment. A user captures an image sequence of a trampoline athlete using a video capture device 501 and uploads the image sequence to a server 502. Server 502 first determines the motion state of a target object in the image sequence based on the images in the image sequence, obtaining a motion state sequence. Then, based on the multiple motion states in a sliding window in the motion state sequence, the server updates the target motion state among the multiple motion states, obtaining an updated motion state that conforms to the kinematic laws of the target object. Then, based on the updated motion state, a segmentation point corresponding to a motion process of the target object is determined in the image sequence. Finally, the image sequence is segmented according to the segmentation point.

[0103] In this embodiment, a method for segmenting an image sequence is provided, and a method and device for segmenting an image sequence are provided. According to multiple motion states of the motion state sequence in a sliding window, the target motion state is updated so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations in the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and thus improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0104] In some optional implementations of this embodiment, the above-mentioned execution entity may also perform the following operations: first, determine the attribute information of the motion process based on the segmented sequence obtained by segmenting the image sequence; then, display the segmented sequence and attribute information through the target display device.

[0105] Continue to refer Figure 6 , shows a schematic diagram of the display interface of the target display device. The target display device can use each key image in the segmented sequence as a representative image to display attribute information of each motion process. The target display device can be a display screen, smartphone, or other device with data display capabilities.

[0106] Taking trampoline exercise as an example, attribute information includes but is not limited to the duration, maximum height, and horizontal offset of the exercise process. The duration represents the duration of a movement process and can be determined based on the acquisition time of the start and end frames of the segmented sequence. The maximum height represents the highest jump height during the movement process and can be determined based on the detection frame when the target object is transitioning from an ascending state to a descending state. The horizontal offset represents the horizontal offset distance of the target object during a movement process and can be determined based on the detection frame of the target object in each image.

[0107] In this implementation, the execution entity can determine and display the attribute information of the motion process, which helps to improve the user's information acquisition efficiency and experience.

[0108] Continue to refer Figure 7 , shows a schematic process 700 of another embodiment of the image sequence segmentation method according to the present disclosure. In the process 700, the following steps are included:

[0109] Step 701 : Based on images in an image sequence, determine the motion state of a target object in the image to obtain a motion state sequence.

[0110] Step 702 : Determine whether the number of motion states that are the same as the target motion state among the multiple motion states exceeds a preset number threshold.

[0111] Step 703 : In response to the determination being yes, judging whether taking the target motion state as the updated motion state complies with the kinematic law based on the historical updated motion state.

[0112] Among them, the historical updated motion state is the updated motion state determined according to the motion state in the historical sliding window; the updated motion state includes the rising state and the falling state corresponding to the jumping motion, and the kinematic law is characterized by preset conditions, which include: the target motion state is different from the updated motion state corresponding to the previous sliding window; and the difference between the acquisition time of the image corresponding to the target motion state and the time when the last state change process occurred exceeds the preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

[0113] Step 704: Based on the judgment result being yes, the target motion state is determined as the updated motion state.

[0114] Step 705 : In response to the number of motion states in the plurality of motion states being the same as the target motion state not exceeding a preset number threshold, determine the motion states in the plurality of motion states whose number exceeds the preset number threshold as updated motion states.

[0115] Step 706 : For an image in the image sequence, in response to the updated motion state corresponding to the image being a descending state and the updated motion state corresponding to the previous frame of the image being an ascending state, determine a segmentation point between the image and the previous frame of the image.

[0116] Step 707: segment the image sequence according to the segmentation points.

[0117] Step 708: Determine the attribute information of the motion process according to the segmented sequence obtained by segmenting the image sequence.

[0118] Step 709: Display the segmented sequence and attribute information via the target display device.

[0119] The process 700 of the method for determining the sequential labels in this embodiment specifically illustrates the process of determining the updated motion state, the process of determining the segmentation points, and the process of displaying the segmented sequence and attribute information, thereby preventing the updated motion state from being disturbed by short-term fluctuations during the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and thereby improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0120] Continue to refer Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image sequence segmentation device. The system embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0121] like Figure 8 As shown, the image sequence segmentation device 800 includes: a state sequence determination unit 801, which is configured to determine the motion state of the target object in the image based on the image in the image sequence to obtain a motion state sequence; a state update unit 802, which is configured to update the target motion state among multiple motion states according to the multiple motion states in the sliding window in the motion state sequence to obtain an updated motion state that conforms to the kinematic law of the target object; a segmentation point determination unit 803, which is configured to determine the segmentation point corresponding to a motion process of the target object in the image sequence according to the updated motion state; and an image sequence segmentation unit 804, which is configured to segment the image sequence according to the segmentation point.

[0122] In some optional implementations of this embodiment, the state update unit 802 is further configured to: determine whether the number of motion states that are the same as the target motion state in multiple motion states exceeds a preset number threshold; in response to determining yes, judge whether taking the target motion state as the updated motion state complies with the kinematic law based on the historical updated motion state, wherein the historical updated motion state is the updated motion state determined based on the motion state in the historical sliding window; based on the judgment result being yes, determine the target motion state as the updated motion state.

[0123] In some optional implementations of this embodiment, the updated motion state includes an ascending state and a descending state corresponding to jumping-type motions, and the kinematic law is characterized by preset conditions, which include: the target motion state is different from the updated motion state corresponding to the previous sliding window; and the difference between the acquisition time of the image corresponding to the target motion state and the time when the last state change process occurred exceeds a preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

[0124] In some optional implementations of this embodiment, the state update unit 802 is further configured to: in response to the number of motion states in multiple motion states that are the same as the target motion state does not exceed a preset number threshold, determine the motion state in multiple motion states whose number exceeds the preset number threshold as the updated motion state.

[0125] In some optional implementations of this embodiment, the segmentation point determination unit 803 is further configured to: for an image in an image sequence, in response to the updated motion state corresponding to the image being a descending state, and the updated motion state corresponding to the previous frame of the image being an ascending state, determine a segmentation point between the image and the previous frame of the image.

[0126] In some optional implementations of this embodiment, the target motion state is the last motion state among multiple motion states; and the state update unit 802 is further configured to: update the last motion state among multiple motion states according to the multiple motion states to obtain an updated motion state.

[0127] In some optional implementations of this embodiment, the apparatus further includes: a full load determination unit (not shown in the figure), configured to determine whether the number of motion states in the sliding window reaches the capacity of the sliding window.

[0128] In some optional implementations of this embodiment, the image is a depth image, and the state sequence determination unit 801 is further configured to: determine a target depth image corresponding to the motion range of the target object from the depth image; determine a detection frame corresponding to the target object from the target depth image; and determine the motion state of the target object based on the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

[0129] In some optional implementations of this embodiment, the above-mentioned device also includes: an attribute determination unit (not shown in the figure), configured to determine the attribute information of the motion process based on the segmented sequence obtained by segmenting the image sequence; and a display unit (not shown in the figure), configured to display the segmented sequence and attribute information through a target display device.

[0130] In this embodiment, a device for segmenting an image sequence is provided, which updates the target motion state in the motion state sequence according to multiple motion states in a sliding window so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations in the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and further improving the real-time performance and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the image sequence segmentation method described in any of the above embodiments when executing.

[0132] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the image sequence segmentation method described in any of the above embodiments when executed.

[0133] An embodiment of the present disclosure provides a computer program product, which, when executed by a processor, can implement the image sequence segmentation method described in any of the above embodiments.

[0134] Figure 9A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0135] like Figure 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0136] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0137] The computing unit 901 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the image sequence segmentation method. For example, in some embodiments, the image sequence segmentation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the image sequence segmentation method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the image sequence segmentation method by any other suitable means (e.g., via firmware).

[0138] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable image sequence segmentation device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0142] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0143] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private servers (VPS). It may also be a server in a distributed system or a server integrated with blockchain.

[0144] According to the technical solution of the embodiments of the present disclosure, a method and device for segmenting an image sequence are provided. According to the multiple motion states of the motion state sequence in the sliding window, the target motion state is updated so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations during the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and thus improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0145] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not a limitation herein.

[0146] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for segmenting an image sequence, comprising: determining, based on images in the image sequence, a motion state of a target object in the image to obtain a motion state sequence; determining whether the number of motion states in the motion state sequence that is the same as the target motion state in the motion state sequence exceeds a preset number threshold; In response to the determination being yes, determining whether taking the target motion state as the updated motion state complies with a kinematic law based on a historical updated motion state, wherein the historical updated motion state is an updated motion state determined based on the motion state in the historical sliding window; Based on the judgment result being yes, determining the target motion state as the updated motion state; determining, in the image sequence, a segmentation point corresponding to a motion process of the target object according to the updated motion state; segmenting the image sequence according to the segmentation points; The target motion state is any one of the multiple motion states, and the target motion states in different sliding windows have the same position in the corresponding sliding windows; The updated motion state includes an ascending state and a descending state corresponding to a jumping motion, and the kinematic law is characterized by a preset condition, which includes: The target motion state is different from the updated motion state corresponding to the previous sliding window; and The difference between the acquisition time of the image corresponding to the target motion state and the occurrence time of the last state change process exceeds a preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

2. The method according to claim 1, wherein Also includes: In response to the number of motion states identical to the target motion state in the multiple motion states not exceeding the preset number threshold, the motion states in the multiple motion states whose number exceeds the preset number threshold are determined as the updated motion states.

3. The method according to claim 2, wherein: Determining, in the image sequence, a segmentation point corresponding to a motion process of the target object according to the updated motion state includes: For an image in the image sequence, in response to the updated motion state corresponding to the image being a descending state and the updated motion state corresponding to the previous frame image of the image being an ascending state, the segmentation point is determined between the image and the previous frame image of the image.

4. The method according to any one of claims 1 to 3, wherein The target motion state is the last motion state among the multiple motion states.

5. The method according to any one of claims 1 to 3, wherein Before determining whether the number of motion states in the motion state sequence that is the same as the target motion state in the motion state sequence exceeds a preset number threshold, the method further includes: It is determined that the number of motion states in the sliding window reaches a capacity of the sliding window.

6. The method according to claim 1, wherein The image is a depth image, and Determining the motion state of the target object in the image includes: Determining a target depth image corresponding to the motion range of the target object from the depth image; Determine a detection frame corresponding to the target object from the target depth image; The motion state of the target object is determined according to the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

7. The method according to any one of claims 1 to 3, wherein Also includes: determining attribute information of the motion process according to a segmented sequence obtained by segmenting the image sequence; The segmented sequence and the attribute information are displayed through a target display device.

8. A device for segmenting an image sequence, comprising: a state sequence determining unit configured to determine, based on images in the image sequence, a motion state of a target object in the image, to obtain a motion state sequence; a state updating unit configured to: determine whether the number of motion states in the motion state sequence that is the same as the target motion state in the motion state sequence exceeds a preset number threshold; In response to the determination being yes, determining whether taking the target motion state as the updated motion state complies with a kinematic law based on the historical updated motion state, wherein the historical updated motion state is an updated motion state determined based on the motion state in the historical sliding window; if the determination result is yes, determining the target motion state as the updated motion state; a segmentation point determination unit configured to determine, in the image sequence, a segmentation point corresponding to a motion process of the target object according to the updated motion state; an image sequence segmentation unit, configured to segment the image sequence according to the segmentation points; The target motion state is any one of the multiple motion states, and the target motion states in different sliding windows have the same position in the corresponding sliding windows; The updated motion state includes an ascending state and a descending state corresponding to a jumping motion, and the kinematic law is characterized by a preset condition, which includes: The target motion state is different from the updated motion state corresponding to the previous sliding window; and The difference between the acquisition time of the image corresponding to the target motion state and the occurrence time of the last state change process exceeds a preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

9. The device according to claim 8, wherein The state updating unit is further configured to: In response to the number of motion states identical to the target motion state in the multiple motion states not exceeding the preset number threshold, the motion states in the multiple motion states whose number exceeds the preset number threshold are determined as the updated motion states.

10. The device according to claim 9, wherein The segmentation point determination unit is further configured to: For an image in the image sequence, in response to the updated motion state corresponding to the image being a descending state and the updated motion state corresponding to the previous frame image of the image being an ascending state, the segmentation point is determined between the image and the previous frame image of the image.

11. The device according to any one of claims 8 to 10, wherein: The target motion state is the last motion state among the multiple motion states.

12. The device according to any one of claims 8 to 10, wherein: Also includes: The full load determination unit is configured to determine whether the number of motion states in the sliding window reaches the capacity of the sliding window.

13. The device according to claim 8, wherein The image is a depth image, and The state sequence determining unit is further configured to: Determine a target depth image corresponding to the motion range of the target object from the depth image; determine a detection frame corresponding to the target object from the target depth image; and determine the motion state of the target object based on the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

14. The device according to any one of claims 8 to 10, wherein: Also includes: an attribute determination unit configured to determine attribute information of the motion process according to a segmented sequence obtained by segmenting the image sequence; The display unit is configured to display the segmented sequence and the attribute information through a target display device.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images