Image sequence segmentation method and device and computer program product

Abstract

The invention provides an image sequence segmentation method and device, electronic equipment, a storage medium and a computer program product, relates to the technical field of artificial intelligence such as deep learning and large models, and can be applied to an intelligent sports scene. According to the specific implementation scheme, based on images in an image sequence, the motion state of a target object in the images is determined, and a motion state sequence is obtained; according to a plurality of motion states in the sliding window in the motion state sequence, updating a target motion state in the plurality of motion states to obtain an updated motion state conforming to the kinematics law of the target object; according to the updated motion state, determining a segmentation point corresponding to a motion process of the target object in the image sequence; and segmenting the image sequence according to the segmentation points.

Description

Technical Field

[0001] The present disclosure relates to artificial intelligence technology fields such as deep learning and large models, and in particular to an image sequence segmentation method, device, electronic device, storage medium and computer program product, which can be applied to scenarios such as smart sports. Background Art

[0002] In today's professional sports field, the demand for high-precision, real-time sports performance analysis is growing. This demand stems from the continuous pursuit of optimizing athlete performance, including accurate analysis of technical details, instant adjustment of sports strategies, and effective management of injury prevention. Traditional analysis methods often rely on manual observation of sports videos and manual post-data processing. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device, storage medium and computer program product for segmenting an image sequence.

[0004] According to a first aspect, a method for segmenting an image sequence is provided, comprising: determining, based on images in the image sequence, a motion state of a target object in the image to obtain a motion state sequence; updating, based on a plurality of motion states in a sliding window in the motion state sequence, a target motion state among a plurality of motion states to obtain an updated motion state that conforms to the kinematic law of the target object; determining, based on the updated motion state, a segmentation point corresponding to a motion process of the target object in the image sequence; and segmenting the image sequence according to the segmentation point.

[0005] According to a second aspect, a device for segmenting an image sequence is provided, comprising: a state sequence determination unit, configured to determine the motion state of a target object in an image based on an image in the image sequence, and obtain a motion state sequence; a state update unit, configured to update a target motion state among multiple motion states according to multiple motion states in a sliding window in the motion state sequence, and obtain an updated motion state that conforms to the kinematic law of the target object; a segmentation point determination unit, configured to determine a segmentation point corresponding to a motion process of the target object in the image sequence according to the updated motion state; and an image sequence segmentation unit, configured to segment the image sequence according to the segmentation point.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a method described in any implementation manner of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect when executed by a processor.

[0009] According to the technology disclosed in the present invention, a method and device for segmenting an image sequence are provided. According to multiple motion states in a sliding window in a motion state sequence, a target motion state is updated so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations in the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and further improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0013] Figure 2 is a flow chart of an embodiment of a method for segmenting an image sequence according to the present disclosure;

[0014] Figure 3 Schematic diagram of image acquisition during trampoline motion according to this embodiment;

[0015] Figure 4 is a schematic diagram of a specific determination process of the updated motion state according to this embodiment

[0016] Figure 5 is a schematic diagram of an application scenario of the image sequence segmentation method according to this embodiment;

[0017] Figure 6 is a schematic diagram of a display interface of a target display device according to this embodiment;

[0018] Figure 7 is a flow chart of an embodiment of a data recommendation method according to the present disclosure;

[0019] Figure 8is a structural diagram of an embodiment of an image sequence segmentation device according to the present disclosure;

[0020] Fig. 9 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 An exemplary architecture 100 is shown to which the image sequence segmentation method and apparatus disclosed herein can be applied.

[0024] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0025] Terminal devices 101, 102, 103 may be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, 103 are hardware, they may be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smart phones, tablet computers, e-book readers, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules for providing distributed services, or they may be implemented as a single software or software module. No specific limitation is made here.

[0026] The server 105 may be a server that provides various services, for example, a background processing server that determines the segmentation points of the image sequence to segment the image sequence for the image sequence provided by the terminal devices 101, 102, and 103. Optionally, the server may feed back the segmented sequence obtained by segmenting the image sequence to the terminal device. As an example, the server 105 may be a cloud server.

[0027] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0028] It should also be noted that the image sequence segmentation method provided in the embodiments of the present disclosure is generally executed by a server, but the possibility of being executed by a terminal device, or being executed by a server and a terminal device in cooperation with each other is not excluded. Accordingly, the various parts (such as various units) included in the image sequence segmentation device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0029] It should be understood that Figure 1 The number of terminal devices, networks and servers in the system is only illustrative. According to the implementation requirements, there may be any number of terminal devices, networks and servers. When the electronic device on which the image sequence segmentation method is running does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (such as a terminal device or a server) on which the image sequence segmentation method is running.

[0030] Please refer to Figure 2 , Figure 2 A flowchart of a method for segmenting an image sequence provided by an embodiment of the present disclosure. In process 200, the following steps are included:

[0031] Step 201: determining the motion state of a target object in an image sequence based on images in the image sequence to obtain a motion state sequence.

[0032] In this embodiment, the execution subject of the image sequence segmentation method (for example, Figure 1 The server in the image sequence can obtain the image sequence remotely or locally through a wired network connection or a wireless network connection, and determine the motion state of the target object in the image based on the images in the image sequence to obtain a motion state sequence.

[0033] The image sequence is an original video obtained by recording the target object's movement process under the authorization of the target object, or a processed video obtained by specific processing (such as enhancing the clarity, filtering key frames, etc.) based on the original video. The target object can be an object with a movement process, such as an athlete or various sports equipment operated by the athlete.

[0034] As an example, the execution subject may input the images in the image sequence into the motion state determination model one by one or in batches, determine the motion state of the target object in each image in the image sequence through the motion state determination model, and combine the motion states of the target objects in each image according to the temporal relationship represented by the image sequence to obtain a motion state sequence. The motion state determination model is used to represent the corresponding relationship between the image and the motion state of the target object in the image, such as a convolutional neural network, a recurrent neural network, a large visual language model, etc.

[0035] As another example, for each image in an image sequence, the above-mentioned execution entity can determine the feature information (such as facial features, limb features) of the target object in the image, and then determine the motion state of the target object in the image based on the similarity between the feature information and the standard feature information of each motion state.

[0036] The target object has different motion states corresponding to different types of sports. For example, in jumping sports such as trampoline, pole vaulting, and high jump, the motion state includes rising state and falling state; in ball sports such as football and basketball, the motion state includes holding the ball (touching the ball) and not holding the ball (not touching the ball).

[0037] In some optional implementations of this embodiment, the images in the image sequence are depth images. A depth image, also known as a range image, refers to an image that uses the distance (depth) from an image acquisition device (such as a camera or a depth sensor) to each point in the scene as a pixel value.

[0038] In this implementation, the execution subject may perform step 201 as follows:

[0039] First, a target depth image corresponding to the motion range of the target object is determined from the depth image.

[0040] In this implementation, the upper depth threshold Dmax and the lower depth threshold Dmin are determined in advance according to the actual situation on site; then the pixels with pixel values ​​between the lower threshold Dmin and the upper threshold Dmax are determined in the depth image to obtain the target depth image corresponding to the motion range of the target object. Figure 3, shows a schematic diagram of image acquisition in trampoline sports. In trampoline sports, the athlete's movement process is on the trampoline, that is, the range of their movement is the area corresponding to the trampoline. Therefore, the distance between the image acquisition device and the near end of the trampoline (the end closer to the image acquisition device) can be used as the lower threshold Dmin, and the distance between the image acquisition device and the far end of the trampoline (the end farther from the image acquisition device) can be used as the upper threshold Dmax.

[0041] Then, the detection box corresponding to the target object is determined from the target depth image.

[0042] The detection frame is the minimum bounding box corresponding to the target object. The above execution subject can binarize the target depth image to determine the detection frame corresponding to the target object.

[0043] Finally, the motion state of the target object is determined based on the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

[0044] As an example, the two-dimensional center coordinate Pos of the target object can be obtained by the center of the detection box BBox coordinate; now The two-dimensional center coordinate Pos at the moment (corresponding to the target depth image) t and T now-1 Time positionPos t-1 , the displacement is calculated by the following formula:

[0045] ΔY=(Pos t -Pos t-1 )×Y

[0046] Among them, Y represents the height. When ΔY>0, it is in an ascending state; when ΔY<0, it is in a descending state.

[0047] In this implementation, a specific method for determining the motion state of a target object in an image is provided, which is closely adapted to the acquisition method and improves the efficiency of determining the motion state.

[0048] Step 202 , according to the multiple motion states in the sliding window in the motion state sequence, the target motion state in the multiple motion states is updated to obtain an updated motion state that complies with the kinematic law of the target object.

[0049] In this embodiment, the execution subject may update the target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence, and obtain an updated motion state that complies with the kinematic law of the target object.

[0050] The capacity of the sliding window can be specifically set according to actual conditions, for example, the capacity is 5. The target motion state can be any motion state of multiple motion states in the sliding window, but the positions of the target motion states in different sliding windows are the same, for example, the target motion state of the current sliding window and the target motion state of the previous sliding window are both in the middle position of the sliding window.

[0051] Kinematic laws are summarized through observation and experiment, and describe the regularity of the position of a person or object in space changing over time. Taking trampoline sports as an example, the five motion states in the sliding window are (rising state, rising state, falling state, rising state, falling state). It can be understood that the rising state and falling state of the athlete will last for a certain period of time and will not change repeatedly in a very short time. The above five motion states indicate that the motion state of the target object changes repeatedly in a very short time, which means that the five motion states in the sliding window include motion states that do not conform to the kinematic laws.

[0052] Here we explain why the motion state does not conform to the laws of kinematics: Taking trampoline exercise as an example, the actual motion state change process is generally that the athlete is in an upward state from the lowest point to the highest point, and in a downward state from the highest point to the lowest point. However, at the highest point, the athlete is relatively still (the height remains unchanged) for a short period of time (for example, 0.1 seconds). During this period, the video acquisition device usually collects multiple frames of images. Taking an image acquisition device with a frame rate of 50 as an example, it collects one frame of image every 0.02 seconds, and 5 frames of image will be collected within 0.1 seconds. For these 5 frames of image, when determining the detection frame of the target object, the position of the determined detection frame may not be completely consistent due to detection errors, resulting in floating changes of the athlete rising and falling at the highest point. This situation can also be called short-term fluctuation interference.

[0053] As an example, the above-mentioned execution entity can determine whether the target motion state among multiple motion states conforms to the kinematic laws based on the multiple motion states in the sliding window; in response to the yes determination, the target motion state is used as the updated motion state; in response to the no determination, the target motion state is adjusted to conform to the kinematic laws based on the multiple motion states to obtain the adjusted motion state.

[0054] As another example, for different sports events, the execution subject or the electronic device connected to the execution subject in communication is provided with preset conditions that characterize the kinematic laws corresponding to the sports event. The execution subject can determine the preset conditions corresponding to the sports event represented by the image sequence, and then determine whether the target motion state meets the preset conditions in combination with the preset conditions and multiple motion states in the sliding window. In response to meeting the preset conditions, the target motion state is used as the updated motion state; in response to determining that it does not meet the preset conditions, the target motion state is adjusted according to the multiple motion states to meet the preset conditions to obtain the adjusted motion state.

[0055] It can be understood that the sliding window slides on the motion sequence, and for each sliding window, an updated motion state of the target motion state within the sliding window will be obtained, thereby obtaining multiple updated motion states; multiple updated motion states are arranged according to the corresponding time series relationship of the motion state sequence to form an updated motion state.

[0056] In some optional implementations of this embodiment, the execution subject may perform step 202 in the following manner:

[0057] First, it is determined whether the number of motion states that are the same as the target motion state among the plurality of motion states exceeds a preset number threshold.

[0058] As an example, whether the number of motion states that are the same as the target motion state in the multiple motion states exceeds a preset number threshold is used to indicate whether the number of motion states that are the same as the target motion state in the multiple motion states exceeds half.

[0059] Then, in response to the determination being yes, it is determined whether taking the target motion state as the updated motion state complies with the kinematic law based on the historical updated motion state.

[0060] The historical updated motion state is an updated motion state determined according to the motion state in the historical sliding window.

[0061] In this implementation, the execution subject can determine the updated motion states corresponding to all historical sliding windows up to the present or a preset number of historical sliding windows up to the present, i.e., the historical updated motion states, and obtain a historical updated motion state sequence. Then, according to the historical updated motion state sequence, it is determined whether taking the target motion state as the updated motion state complies with the kinematic law.

[0062] Continuing with the trampoline exercise as an example, with the target motion state as the updated motion state, it is determined whether a short-term fluctuation occurs, thereby judging whether taking the target motion state as the updated motion state conforms to the kinematic law.

[0063] Finally, based on the judgment result being yes, the target motion state is determined as the updated motion state.

[0064] When it is determined that taking the target motion state as the updated motion state complies with the kinematic law, the target motion state is taken as the updated motion state.

[0065] In this implementation, a specific method for determining the updated motion state is provided, which improves the accuracy of the determined updated motion state.

[0066] In some optional implementations of this embodiment, the updated motion state includes an ascending state and a descending state corresponding to a jumping motion, and the kinematic law is characterized by preset conditions, which include:

[0067] The target motion state is different from the updated motion state corresponding to the previous sliding window; the difference between the acquisition time of the image corresponding to the target motion state and the time of the last state change process exceeds a preset time difference threshold. The state change process represents the process of change between adjacent updated motion states.

[0068] For jumping motion, the moment of interference by short-term fluctuations is generally when the target object is at its highest or lowest point. At this time, the actual motion state of the target object generally changes, and the difference between this change and the last state change is large.

[0069] Taking the highest point as an example, at this time, a change from an ascending state to a descending state generally occurs. The time when this state change process occurs, that is, the time when the target object is at the highest point during its movement, can generally be represented by the acquisition time of the image corresponding to the highest point, and should be separated from the change time from the ascending state to the descending state in the previous trampoline process by a long time (exceeding the preset time difference threshold).

[0070] The preset time difference threshold can be specifically set according to actual conditions and is not limited here.

[0071] In this implementation, preset conditions for characterizing kinematic laws are specifically provided for jumping movements, which helps to improve the efficiency and accuracy of the process of determining whether the updated motion state conforms to the kinematic laws based on the target motion state.

[0072] In some optional implementations of the present embodiment, the execution entity may also perform step 202 in the following manner: in response to the number of motion states in a plurality of motion states that are the same as the target motion state not exceeding a preset number threshold, determining the motion state in a plurality of motion states that exceeds the preset number threshold as an updated motion state.

[0073] Taking trampoline sports as an example, the five sports states in the sliding window are (rising state, rising state, falling state, rising state, falling state). The target sports state is the "falling state" in the middle position, and its number is 2, which does not exceed the preset number threshold. Then the "rising state" that exceeds the preset number threshold is used as the updated sports state corresponding to the target sports state.

[0074] In this implementation, a method for determining the updated motion state when the preset number threshold is not exceeded is provided, thereby improving the accuracy and determination efficiency of the updated motion state.

[0075] In some optional implementations of this embodiment, the target motion state is the last motion state among the multiple motion states.

[0076] In this implementation, the execution subject may perform step 202 in the following manner: updating the last motion state among the multiple motion states according to the multiple motion states to obtain an updated motion state.

[0077] In this implementation, the target motion state is the last motion state among multiple motion states. The updated motion state can be obtained by determining the last motion state among multiple motion states according to the multiple motion states through the various determination methods of the updated motion state mentioned above.

[0078] In this implementation, the target motion state is set as the last motion state among multiple motion states, and an updated motion state corresponding to the last motion state is determined according to the multiple motion states, which helps to further improve the accuracy of the updated motion state.

[0079] In some optional implementations of this embodiment, before executing the above step 202, the above execution subject may further perform the following operation: determining whether the number of motion states in the sliding window reaches the capacity of the sliding window.

[0080] That is, when the number of motion states in the sliding window reaches the capacity of the sliding window, the operation of determining the updated motion state is not performed.

[0081] Taking the capacity of the sliding window as 5 as an example, in the initial stage, the first to fifth frames of the image sequence are sent to the sliding window in sequence, until the fifth frame, the above-mentioned execution subject will execute the above-mentioned step 202 to obtain the updated motion state; corresponding to the first four frames of the image, the above-mentioned execution subject will not execute the above-mentioned step 202.

[0082] In this implementation, the determination operation of the updated motion state is performed only when the number of motion states in the sliding window accumulates to the capacity of the sliding window, which helps to further improve the accuracy of the updated motion state.

[0083] Continue to refer Figure 4 , which shows a schematic diagram of a specific determination process of the updated motion state.

[0084] The input signal of the determination system that defines the updated motion state (motion state in the motion state sequence) S in And the output signal (updated motion state) S out They are all Boolean variables, which have only two possibilities: positive and negative, corresponding to the rising state and the falling state respectively. The sliding window length is N, which is an odd number.

[0085] 1. Initialization: In the initial stage of motion analysis, the system has no existing state. Whenever the input signal S in , the accumulated signal S in To the sliding window, no signal is output before accumulating N input signals. When accumulating to N frames, S stable Set to more than N / 2 signals, set the time T when the state change process occurs stable is 0.

[0086] 2. Main loop: After the input signal accumulates to N, the sliding window continues to slide along the motion state sequence. in , do the following:

[0087] 2.1. If the N motion states in the sliding window are consistent with the signal S in If the number of identical motion states does not exceed N / 2, the system is in an "unstable state" and the signal S stable Determined as S in The corresponding updated motion state.

[0088] 2.2 If the N motion states in the sliding window are consistent with the signal S in If the number of identical motion states exceeds N / 2, the system is in a "stable state" and the current S stable ; if S in Not equal to S stable And the current time T now Distance T stable The time is greater than the preset time difference threshold T th , then the signal S stable Updated to S in ,T stable Set to T now , and S in As the updated motion state.

[0089] Step 203: Determine a segmentation point corresponding to a motion process of the target object in the image sequence according to the updated motion state.

[0090] In this embodiment, the execution subject can update the post-motion state and determine the segmentation point corresponding to a motion process of the target object in the image sequence. The image sequence often represents the repeated motion process of the target object, and the image sequence between adjacent segmentation points represents a motion process of the target object.

[0091] As an example, the execution subject may analyze each updated motion state in the updated motion state sequence to determine the start time and the end time of a motion process of the target object; and then use the start time and the end time as segmentation points.

[0092] As another example, for different sports, the execution subject can determine the corresponding segmentation point determination method. For example, for jumping sports, the change moment between the rising state and the falling state is used as the segmentation point; for ball sports, the change moment between the holding state and the non-holding state is used as the segmentation point. In this way, the execution subject can determine the segmentation point corresponding to a motion process of the target object in the image sequence according to the segmentation point determination method corresponding to the sports represented by the updated motion state sequence and the image sequence.

[0093] It can be understood that the images in the image sequence and the motion states in the motion state sequence are in one-to-one correspondence; after the number of motion states in the sliding window accumulates to the capacity of the sliding window, the motion states in the motion state sequence and the updated motion states in the updated motion state sequence are in one-to-one correspondence.

[0094] In some optional implementations of the present embodiment, the execution entity may perform step 203 as follows: for an image in an image sequence, in response to the updated motion state corresponding to the image being a descending state, and the updated motion state corresponding to the previous frame of the image being an ascending state, a segmentation point is determined between the image and the previous frame of the image.

[0095] Continue to refer to the above Figure 4 The corresponding implementation method, when S occurs t ! =S t-1 , it indicates that the jump state has changed:

[0096] If S t is in the descending state, the target object is changing from the ascending state to the descending state. At this time, T now At the highest point of the jump;

[0097] If S t is in the rising state, the target object is changing from the falling state to the rising state. At this time, T now Being at the lowest point of the jump, that is, the end of this jump and the beginning of the next jump.

[0098] In this implementation, a segmentation point determination method for jump-type motion is provided, which improves the accuracy and efficiency of the segmentation point determination operation.

[0099] Step 204: segment the image sequence according to the segmentation points.

[0100] In this embodiment, the execution entity may segment the image sequence according to the segmentation points.

[0101] As an example, the execution subject divides the image sequence according to the segmentation points to obtain multiple segmented sequences. Each segmented sequence represents a movement process of the target object.

[0102] Continue to see Figure 5 , Figure 5 500 is a schematic diagram of an application scenario of the image sequence segmentation method according to the present embodiment. The user captures an image sequence of a trampoline athlete through a video capture device 501, and uploads the image sequence to a server 502. The server 502 first determines the motion state of the target object in the image based on the images in the image sequence, and obtains a motion state sequence; then, according to the multiple motion states in the sliding window in the motion state sequence, updates the target motion state in the multiple motion states, and obtains an updated motion state that conforms to the kinematic law of the target object; then, according to the updated motion state, determines a segmentation point corresponding to a motion process of the target object in the image sequence; finally, the image sequence is segmented according to the segmentation point.

[0103] In the present embodiment, a method for segmenting an image sequence is provided, and a method and device for segmenting an image sequence are provided. According to multiple motion states of a motion state sequence in a sliding window, a target motion state is updated so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations in the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and further improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0104] In some optional implementations of this embodiment, the above-mentioned execution subject may also perform the following operations: first, determine the attribute information of the motion process according to the segmented sequence obtained by segmenting the image sequence; then, display the segmented sequence and the attribute information through the target display device.

[0105] Continue to refer Figure 6 , shows a schematic diagram of the display interface of the target display device. The target display device can use each key image in the segmented sequence as a representative image to display the attribute information of each motion process. The target display device can be a display screen, a smart phone, or other device with data display function.

[0106] Taking trampoline exercise as an example, the attribute information includes but is not limited to the duration, maximum height, horizontal offset, etc. of the exercise process. The duration represents the duration of a movement process, which can be determined based on the acquisition time of the start frame image and the end frame image of the segmented sequence. The maximum height represents the highest jump height in the movement process, which can be determined based on the detection frame when the target object is changing from an ascending state to a descending state. The horizontal offset represents the horizontal offset distance of the target object in a movement process, which can be determined based on the detection frame of the target object in each image.

[0107] In this implementation, the above-mentioned execution subject can determine and display the attribute information of the movement process, which helps to improve the user's information acquisition efficiency and experience.

[0108] Continue to refer Figure 7 , shows a schematic process 700 of another embodiment of the image sequence segmentation method according to the present disclosure. In the process 700, the following steps are included:

[0109] Step 701: Based on the images in the image sequence, determine the motion state of the target object in the image to obtain a motion state sequence.

[0110] Step 702: Determine whether the number of motion states that are the same as the target motion state among the multiple motion states exceeds a preset number threshold.

[0111] Step 703 , in response to the determination being yes, judging whether taking the target motion state as the updated motion state complies with the kinematic law based on the historical updated motion state.

[0112] Among them, the historical updated motion state is the updated motion state determined according to the motion state in the historical sliding window; the updated motion state includes the rising state and the falling state corresponding to the jumping motion, and the kinematic law is characterized by preset conditions, and the preset conditions include: the target motion state is different from the updated motion state corresponding to the previous sliding window; and the difference between the acquisition time of the image corresponding to the target motion state and the occurrence time of the last state change process exceeds the preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

[0113] Step 704: based on the judgment result being yes, determine the target motion state as the updated motion state.

[0114] Step 705 , in response to the number of motion states in the multiple motion states being the same as the target motion state not exceeding a preset number threshold, determining the motion states in the multiple motion states whose number exceeds the preset number threshold as updated motion states.

[0115] Step 706 , for an image in the image sequence, in response to the updated motion state corresponding to the image being a falling state and the updated motion state corresponding to the previous frame of the image being an rising state, determine a segmentation point between the image and the previous frame of the image.

[0116] Step 707, segment the image sequence according to the segmentation points.

[0117] Step 708: Determine the attribute information of the motion process according to the segmented sequence obtained by segmenting the image sequence.

[0118] Step 709: Display the segmented sequence and attribute information through the target display device.

[0119] The process 700 of the method for determining the sequential labels in this embodiment specifically illustrates the process of determining the updated motion state, the process of determining the segmentation points, and the process of displaying the segmented sequence and attribute information, thereby preventing the updated motion state from being disturbed by short-term fluctuations during the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and thereby improving the real-time and accuracy of the segmentation process for the image sequence corresponding to the motion state sequence.

[0120] Continue to refer Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image sequence segmentation device. The system embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0121] like Figure 8 As shown, the image sequence segmentation device 800 includes: a state sequence determination unit 801, which is configured to determine the motion state of the target object in the image based on the image in the image sequence to obtain a motion state sequence; a state update unit 802, which is configured to update the target motion state among multiple motion states according to multiple motion states in the sliding window in the motion state sequence to obtain an updated motion state that conforms to the kinematic law of the target object; a segmentation point determination unit 803, which is configured to determine the segmentation point corresponding to a motion process of the target object in the image sequence according to the updated motion state; and an image sequence segmentation unit 804, which is configured to segment the image sequence according to the segmentation point.

[0122] In some optional implementations of the present embodiment, the state update unit 802 is further configured to: determine whether the number of motion states that are the same as the target motion state in multiple motion states exceeds a preset number threshold; in response to determining yes, determine whether it complies with the kinematic law to use the target motion state as the updated motion state based on the historical updated motion state, wherein the historical updated motion state is the updated motion state determined based on the motion state in the historical sliding window; based on the judgment result being yes, determine the target motion state as the updated motion state.

[0123] In some optional implementations of the present embodiment, the updated motion state includes a rising state and a falling state corresponding to a jumping motion, and the kinematic law is characterized by preset conditions, which include: the target motion state is different from the updated motion state corresponding to the previous sliding window; and the difference between the acquisition time of the image corresponding to the target motion state and the time when the last state change process occurred exceeds a preset time difference threshold, wherein the state change process characterizes the process of changes between adjacent updated motion states.

[0124] In some optional implementations of the present embodiment, the state update unit 802 is further configured to: in response to the number of motion states in a plurality of motion states that are the same as the target motion state not exceeding a preset number threshold, determine the motion state in a plurality of motion states whose number exceeds the preset number threshold as an updated motion state.

[0125] In some optional implementations of the present embodiment, the segmentation point determination unit 803 is further configured to: for an image in an image sequence, in response to the updated motion state corresponding to the image being a descending state, and the updated motion state corresponding to the previous frame of the image being an ascending state, determine a segmentation point between the image and the previous frame of the image.

[0126] In some optional implementations of this embodiment, the target motion state is the last motion state among multiple motion states; and the state updating unit 802 is further configured to: update the last motion state among multiple motion states according to the multiple motion states to obtain an updated motion state.

[0127] In some optional implementations of this embodiment, the above-mentioned device also includes: a full load determination unit (not shown in the figure), which is configured to determine whether the number of motion states in the sliding window reaches the capacity of the sliding window.

[0128] In some optional implementations of the present embodiment, the image is a depth image, and the state sequence determination unit 801 is further configured to: determine a target depth image corresponding to the motion range of the target object from the depth image; determine a detection frame corresponding to the target object from the target depth image; and determine the motion state of the target object based on the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

[0129] In some optional implementations of this embodiment, the above-mentioned device also includes: an attribute determination unit (not shown in the figure), configured to determine the attribute information of the motion process based on the segmented sequence obtained by segmenting the image sequence; a display unit (not shown in the figure), configured to display the segmented sequence and attribute information through a target display device.

[0130] In the present embodiment, a device for segmenting an image sequence is provided, which updates a target motion state in a motion state sequence according to multiple motion states in a sliding window so that the updated motion state conforms to the kinematic law, thereby preventing the updated motion state from being disturbed by short-term fluctuations in the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and further improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the image sequence segmentation method described in any of the above embodiments when executing.

[0132] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the image sequence segmentation method described in any of the above embodiments when executed.

[0133] The embodiments of the present disclosure provide a computer program product. When the computer program is executed by a processor, the image sequence segmentation method described in any of the above embodiments can be implemented.

[0134] Fig. 9A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0135] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0136] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0137] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the segmentation method of the image sequence. For example, in some embodiments, the segmentation method of the image sequence may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the segmentation method of the image sequence described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the segmentation method of the image sequence in any other appropriate manner (e.g., by means of firmware).

[0138] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0139] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable image sequence segmentation device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0140] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0142] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0143] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services; it may also be a server for a distributed system, or a server combined with a blockchain.

[0144] According to the technical solution of the embodiments of the present disclosure, a method and device for segmenting an image sequence are provided. According to multiple motion states of the motion state sequence in a sliding window, a target motion state is updated so that the updated motion state conforms to the kinematic law, thereby avoiding the updated motion state from being disturbed by short-term fluctuations in the motion process, improving the accuracy of the segmentation points determined based on the updated motion state, and further improving the real-time and accuracy of the segmentation process of the image sequence corresponding to the motion state sequence.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for segmenting an image sequence, comprising: Based on the images in the image sequence, determining the motion state of the target object in the image to obtain a motion state sequence; According to a plurality of motion states in the sliding window in the motion state sequence, a target motion state in the plurality of motion states is updated to obtain an updated motion state that conforms to the kinematic law of the target object; Determining, in the image sequence, a segmentation point corresponding to a motion process of the target object according to the updated motion state; The image sequence is segmented according to the segmentation points.

2. The method according to claim 1, wherein: The method of updating a target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence to obtain an updated motion state that conforms to the kinematic law of the target object includes: Determining whether the number of motion states that are the same as the target motion state among the multiple motion states exceeds a preset number threshold; In response to the determination being yes, judging whether taking the target motion state as the updated motion state complies with a kinematic law based on the historical updated motion state, wherein the historical updated motion state is an updated motion state determined based on the motion state in the historical sliding window; Based on the judgment result being yes, the target motion state is determined as the updated motion state.

3. The method according to claim 2, wherein: The updated motion state includes an ascending state and a descending state corresponding to a jumping motion, and the kinematic law is characterized by a preset condition, and the preset condition includes: The target motion state is different from the updated motion state corresponding to the previous sliding window; and The difference between the acquisition time of the image corresponding to the target motion state and the occurrence time of the last state change process exceeds a preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

4. The method according to claim 2, wherein: The method of updating the target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence to obtain the updated motion state that conforms to the kinematic law of the target object also includes: In response to the number of motion states in the multiple motion states being the same as the target motion state not exceeding the preset number threshold, the motion states in the multiple motion states whose number exceeds the preset number threshold are determined as the updated motion states.

5. The method according to claim 4, wherein: The step of determining, in the image sequence according to the updated motion state, a segmentation point corresponding to a motion process of the target object comprises: For an image in the image sequence, in response to the updated motion state corresponding to the image being a descending state and the updated motion state corresponding to the previous frame of the image being an ascending state, the segmentation point is determined between the image and the previous frame of the image.

6. The method according to any one of claims 1 to 5, wherein: The target motion state is the last motion state among the multiple motion states; and The method of updating a target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence to obtain an updated motion state that conforms to the kinematic law of the target object includes: The last motion state among the multiple motion states is updated according to the multiple motion states to obtain the updated motion state.

7. The method according to any one of claims 1 to 5, wherein: Before updating the target motion state among the multiple motion states according to the multiple motion states in the sliding window in the motion state sequence to obtain the updated motion state that conforms to the kinematic law of the target object, the method further includes: It is determined that the number of motion states in the sliding window reaches a capacity of the sliding window.

8. The method according to claim 1, wherein: The image is a depth image, and Determining the motion state of the target object in the image includes: Determine a target depth image corresponding to the motion range of the target object from the depth image; Determine a detection frame corresponding to the target object from the target depth image; The motion state of the target object is determined according to the relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

9. The method according to any one of claims 1 to 8, wherein: Also includes: determining attribute information of the motion process according to a segmented sequence obtained by segmenting the image sequence; The segmented sequence and the attribute information are displayed through a target display device.

10. A device for segmenting an image sequence, comprising: a state sequence determining unit, configured to determine the motion state of the target object in the image based on the image in the image sequence, and obtain a motion state sequence; a state updating unit configured to update a target motion state among the plurality of motion states according to a plurality of motion states in the sliding window in the motion state sequence, so as to obtain an updated motion state conforming to the kinematic law of the target object; a segmentation point determination unit, configured to determine, in the image sequence, a segmentation point corresponding to a motion process of the target object according to the updated motion state; The image sequence segmentation unit is configured to segment the image sequence according to the segmentation points.

11. The device according to claim 10, wherein: The state updating unit is further configured to: Determine whether the number of motion states that are the same as the target motion state in the multiple motion states exceeds a preset number threshold; in response to the determination being yes, judge whether it complies with the kinematic law to take the target motion state as the updated motion state based on the historical updated motion state, wherein the historical updated motion state is an updated motion state determined based on the motion state in the historical sliding window; based on the judgment result being yes, determine the target motion state as the updated motion state.

12. The device according to claim 11, wherein The updated motion state includes an ascending state and a descending state corresponding to a jumping motion, and the kinematic law is characterized by a preset condition, and the preset condition includes: The target motion state is different from the updated motion state corresponding to the previous sliding window; and The difference between the acquisition time of the image corresponding to the target motion state and the occurrence time of the last state change process exceeds a preset time difference threshold, wherein the state change process represents the process of change between adjacent updated motion states.

13. The device according to claim 11, wherein: The state updating unit is further configured to: In response to the number of motion states in the multiple motion states being the same as the target motion state not exceeding the preset number threshold, the motion states in the multiple motion states whose number exceeds the preset number threshold are determined as the updated motion states.

14. The device according to claim 13, wherein: The segmentation point determination unit is further configured to: For an image in the image sequence, in response to the updated motion state corresponding to the image being a descending state and the updated motion state corresponding to the previous frame of the image being an ascending state, the segmentation point is determined between the image and the previous frame of the image.

15. The device according to any one of claims 10 to 14, wherein: The target motion state is the last motion state among the multiple motion states; and The state updating unit is further configured to: The last motion state among the multiple motion states is updated according to the multiple motion states to obtain the updated motion state.

16. The device according to any one of claims 10 to 14, wherein: Also includes: The full load determination unit is configured to determine whether the number of motion states in the sliding window reaches the capacity of the sliding window.

17. The device according to claim 10, wherein: The image is a depth image, and The state sequence determination unit is further configured to: Determine a target depth image corresponding to the motion range of the target object from the depth image; determine a detection frame corresponding to the target object from the target depth image; and determine the motion state of the target object based on a relative position relationship between the detection frame of the target depth image and the detection frame corresponding to the previous frame image.

18. The device according to any one of claims 10 to 17, wherein: Also includes: an attribute determination unit configured to determine attribute information of the motion process according to a segmented sequence obtained by segmenting the image sequence; A display unit is configured to display the segmented sequence and the attribute information through a target display device.

19. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images